Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
I think when we apply math, we see that it takes different reasoning steps.
We see this kind of different types of changes than you see at SFT and DPO.
I do think it
there is gonna be a lot more understanding of like what these stages actually do.
But it just does really feel like when you do these and when you look at the loss functions, so there's kind of more that you can change about the weights.
doing RL or doing this kind of direct preference optimization optimization sort of thing.
And
That's something that we've heard for a while.
I think leaders in RLHF have said like RL is just a more flexible loss function.
There's a lot more sc scale you can do.
I think that was like the framing that somebody like Jason Way at OpenAI gave in a talk is like, look, like RL has a lot more leg room because you can just the loss function is very different and we haven't explored it.
And things like O one and kind of our own experiments make me more optimistic in this.
It's just like it's very weird.
You can do a lot of weird things.
It
changes the model, notably the vowel scores don't go down that much and it will take a lot more intuition both on the individual level and the academic level to like understand these parameter changes.
RL is doing something very different than the previous loss function.
In some ways it's remarkable that you can just do SFT and pre-training and then you can just throw RL at it.
And the the reg the KL regularization is enough to just like hold it in place.
But like you just change loss functions ninety nine percent of the way there and it doesn't break everything.
Showing 381–400 of 1,013 · page 20 of 51
← Previous
Next →