Nathan Lambert

speaker

1,814 appearances 3 recordings 2 series first heard Feb 2025 last heard 1 Feb

Nathan Lambert’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.

Appearances

newest first · ▶ plays the moment

Lex Fridman Podcast · #490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI · 1 Feb 2026 podcast

makes more sense with the problem formulation, which is why it kind of ends up all forming together.

But to summarize, it's like mid-training is give the model the skills it needs to then learn.

RL and verifiable rewards is let the model try a lot of times.

So put a lot of compute into trial and error learning across hard problems.

And then RLHF would be like, finish the model, make it easy to use, and kind of just round the model out.

It's only gotten up and up.

So I think Grok4 was famous for saying they use a similar amount of compute for pre-training and post-training.

Back to the scaling discussion, they involve very different hardware for scaling.

Pre-training is very compute bound, which is like this flops discussion, which is just how many matrix multiplications can you get through in one time?

And because RL, you're generating these answers, you're trying the model in the real world environments, it ends up being much more memory bound because you're generating long sequences and the attention mechanisms have this

behavior where you get a quadratic increase in memory as you're getting to longer sequences.

So the compute becomes very different.

So when in pre-training, we would talk about a model.

I think if we go back to like the Biden administration executive order, it's like 10 to the 25th flops to train a model.

If you're using flops in post-training, it's a lot weirder because the reality is just like, how many hours are you allocating?

How many GPUs for?

And I think in terms of time, the RL compute is getting much closer because you just can't put it all into one system.

Pre-training is so computationally dense where all the GPUs are talking to each other and it's extremely efficient, where RL has all these moving parts and it can just take a long time to generate a sequence of 100,000 tokens.

If you think about GPT-5.2 Pro taking an hour, it's like, what if your training run has a sample for an hour and you have to make it so that's handled efficiently?

So I think in GPU hours or just like wall clock hours, the RL runs are probably approaching the number of days as pre-training, but they probably aren't using as many GPUs at the same time.

Showing 521–540 of 1,814 · page 27 of 91