Nathan Lambert

speaker

1,814 appearances 3 recordings 2 series first heard Feb 2025 last heard 1 Feb

Nathan Lambert’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.

Appearances

newest first · ▶ plays the moment

Lex Fridman Podcast · #490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI · 1 Feb 2026 podcast

Quen is documented going up to like 50 trillion.

And there's rumors that these closed labs can go to like 100 trillion tokens.

And just getting this potential data to put in, I think they have a very big funnel.

And then the data you actually train the model on is a small percentage of this.

Like the

So this character recognition data would be described as synthetic data for pre-training in a lab.

And then there's also the things like chat GPT now gets wonderful answers and you can train on those best answers.

And that's synthetic data.

It's very different than like early chat GPT, lots of hallucinations data when people became grounded in synthetic data.

I think if we had more compute, we would train for longer.

I think we ultimately see that as a

just something we would want to do.

And especially with big models, you need to have more compute because we talk about having more parameters and we talk about knowledge.

And essentially, there's a ratio where big models can absorb more from data and you get more benefit out of this.

It's like one of these, any logarithmic graph in your mind is like a small model will level off sooner if you're measuring tons of tokens and bigger models need more.

But mostly, we aren't training that big of models right now at AI2 and getting the highest quality data we can is...

the natural starting point.

So I think historically in the open, there's been like a canonical best pre-training data set that has moved around between who has the most recent one or the best recent effort.

Like AI2's Dolmo was very early with the first Olmo and Hugging Face had FineWeb.

And there's a DCLM project, which has been kind of like a... Which is...

Showing 321–340 of 1,814 · page 17 of 91