Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
Plus like have some more fine grained control over the generation, you're like, oh it kinda you squint, it's like you just have to figure out how to put these things together.
The right intuition is that the reward model is trained to do PPO, but in the RL framing, the reward model is actually the environment in a way.
So the environment RL is supposed to be what returns the reward.
So the reward model is a very constrained environment and your actions your inputs to the environment are prompts and your actions are completions.
So it's like a s totally broken RL environment, but the reward model is
not part of like the training update.
It's more of a isolated thing that we just happen to train.
Do.
I've talked to them a lot.
I feel a lot I feel like I had more of an intuition for this like six months ago when I was really in vogue, but
It I do think like it can s it's still changing the tokens individually.
Like it it's like it it is in that way doing something dem similar, but is not doing it through a value model.
It's only mediated through the loss, which looks at the sums of log problems.
So this kind of removes that intermediate stub.
that's
Uh yeah, I mean uh in our setup we tried to throw PPO in at the end and we haven't gotten PPO to beat our DPO setup again.
So it's like, okay, we went through this big process where we saw PPO is better.
And I still think like if we kept hammering at it, we could probably make PPO better.
But just experimentation time and data is so much more important that it's just like
We're not gonna take f four times the compute and experimental time to it's not worth it.
Showing 521–540 of 1,013 · page 27 of 51
← Previous
Next →