Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
Like you need to have that somewhere to kind of kickstart things.
I think pre-trained data probably has some of it, but I think that that's
a balance that we'll see evolve, which is like what is the basic amount that you need to do some instruction following?
And then like how much more do you need for specific capabilities, I think.
Um at some point a good thing for that I did many times saying this project is good for the reader is like go look at the Llama report and look at the percentage of instances per domain
at instruction tuning.
And at instruction tuning, they have way higher math coding reasoning than preference tuning.
So I think like those are the domains where the model needs like just more flops to understand the basic capabilities in SFT.
But like at preference tuning it's much more of
it becomes a contrastive loss function either through GPO, which is like pairs in comparison them.
If you dig into the DPO math it's like actually some weird double negative where it's like
decreasing the probability of the negative of the rejected response rather it like does that more than increasing the probability of the chosen there's weird oddities in the actual DPO.
Math there.
If you go really deep.
PPO is almost more intuitive where you you generate new samples, you have a value model that assigns attribution to each of the tokens where higher number is good, and then it tries to increase the likelihood of things that it sees as being good.
And in this case, it's guided by a reward model.
In our case, it's guided by like, is the answer right?
And
it becomes much more flexible.
I think like DPO is somewhat restricted to the generations that you give it, but RL in that way is flexible where I think it just can kind of change the behavior of the model more.
Showing 361–380 of 1,013 · page 19 of 51
← Previous
Next →