Nathan Lambert

speaker

1,013 appearances 1 recordings 1 series first heard Nov 2024 last heard Nov 2024

Nathan Lambert’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

No recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.

Appearances

newest first · ▶ plays the moment

"The Cognitive Revolution" · Everything You Wanted to Know About LLM Post-Training, with Nathan Lambert of Allen Institute for AI · 21 Nov 2024 podcast

So it's like we're really, really not changing the model very much doing this RL for math compared to what you would do if you were doing PPO for everything.

But if you also compare the that to like best event sampling or something, best event sampling over a reward model also has way lower KL spend than doing this full like PPO online RL thing.

And I don't know the intuitions off the top of my head for where DPO falls.

But it's like all of these preference that's like kind of a way to measure how much you're changing the model in preference tuning.

It's like what is the KL difference across a controlled set of prompts.

There's a problem where we're in all of these numbers that I'm saying it's on a different set of training prompts.

So it's almost like we need to have like a standard of like these are the prompts that we evalu these hundred prompts are what we evaluate our KL distances on across different domains to really see how much the model is moving in general.

But I think that's sufficient.

That's a good way t that people can look at it.

time.

There are people that there are experiments where you can do like do one round of PPO.

with your reference as the SFT model.

And then you can like start another one with your reference as the

first PPO model or DPO.

So people do do things like that.

It's just not as popular.

There's definitely some papers there, which is like moving like resetting your DPO reference to give you more ability to learn.

I don't remember the names off the top of my head, but that is a sort of idea that people

Fiddle with.

This is like what is there's an approximation that you use in a lot of

Showing 421–440 of 1,013 · page 22 of 51