Nathan Lambert

speaker

1,814 appearances 3 recordings 2 series first heard Feb 2025 last heard 1 Feb

Nathan Lambert’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.

Appearances

newest first · ▶ plays the moment

Lex Fridman Podcast · #490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI · 1 Feb 2026 podcast

It stands for data comp language model.

There's been data comp for other machine learning projects and they have had a very strong data set.

And a lot of it is the internet is becoming fairly closed off.

So we have common crawl, which I think is hundreds of trillions of tokens and you filter it.

And it looks like being a lot of scientific work where you're training classifiers and making decisions based on how do you prune down this, this data set into the highest quality stuff and the stuff that suits your tasks.

So previously language models were tested a lot more on knowledge and just kind of conversational things, but now they're expected to do math and code.

So to train a reasoning model, you need to remix your whole dataset.

And there's a lot of actually wonderful scientific methods here where you can take your gigantic dataset, you sample a lot of really tiny things from different sources.

So you say you have GitHub,

Stack Exchange, Reddit, Wikipedia, you can sample small things from them and you train small models on each of these mixes and measure their performance on your evaluations.

And you can just do like basic linear regression and it's like, here's your optimal dataset.

But if your evaluations change, your dataset changes a lot.

So a lot of OMO3 was new sources for reasoning to be better at math and code.

And then you do this mixing procedure and it gives you the answer.

And I think that's a lot of that's happened at labs this year.

It's like there's new hot things, whether it's like

coding environments or web navigation.

You just need to bring in new data.

You need to change your whole pre-training so your post-training can work better and stuff like this.

And that's like the constant re-evolution and the re-determining of what they care about for their models.

Showing 341–360 of 1,814 · page 18 of 91