Nathan Lambert
speaker
1,814 appearances
3 recordings
2 series
first heard Feb 2025
last heard 1 Feb
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 2 in all, peaking in Feb 2026 with 1.
Appearances
Other architectures then using different types of attention.
We're also talking about mixture of experts models.
This sparse nature of MOE models makes it much more efficient to do generation, which becomes a big part of post-training.
And it's like, you need to have your architecture ready so that you can actually scale up this compute.
I still think most of the compute is going in at pre-training because you can still make a model better.
You still want to go and revisit this.
You still want the best base model that you can.
And in a few years, that'll saturate and the RL compute will just go longer.
People vibe that way and describe it in that way, but I think it's not the practice that is happening.
The excitement is elsewhere.
So the low-hanging fruit in RL is elsewhere.
Like, for example, we released our model in November for every company has deadlines.
Our deadline was like November 20th.
And for that, our RL run was five days, which compared to...
2024 is a very long time to just be doing post-training at a model of like 30 billion parameters it's not a big model and then in december we had another release which was just we let the rl run for go go for another three and a half weeks and the model got notably better so we release it and like that's a big amount of time to just allocate to like something that is going to be your peak for the year so it's like so reasoning types of decisions that happen when they're training a model where they just like can't they can't leave it forever you have to keep
You have to keep pulling in the improvements you have from your researchers.
So that's like you redo pre-training.
You'll do this post-training for a month, but then you need to give it to your users.
You need to do safety testing.
So it was kind of just like, I think there's a lot in place that reinforces this cycle of just keep updating the models.
Showing 261–280 of 1,814 · page 14 of 91
← Previous
Next →