Nathan Lambert

speaker

1,013 appearances 1 recordings 1 series first heard Nov 2024 last heard Nov 2024

Nathan Lambert’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

No recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.

Appearances

newest first · ▶ plays the moment

"The Cognitive Revolution" · Everything You Wanted to Know About LLM Post-Training, with Nathan Lambert of Allen Institute for AI · 21 Nov 2024 podcast

Yeah, so the I guess I'm gonna jump to the real nuanced thing, which you might not even have in your questions.

It's like what are open recipes doing and what are closed companies doing?

Um in the open we have the benefit of being able to train on outputs from very strong models like GPT four, which is something that we did.

It's there's

Some amount of legal uncertainty there, but we definitely put it the wording in our data sets is like look, your understanding, your training on outputs from these models.

Um, but that definitely gives an advantage in some way.

Llama, in order to get a model like GPT-4, needs to build Llama 405B.

They need to do instruction tuning over their various base models and all of these things that kind of get this distillation model to work from.

So they are trying to do a similar thing, which is a lot of the instructions are written by a very strong model.

So a lot of math and code instruction data is now written by for LAM.

Llama 405B for open AI, GPT models.

And in that way, we are doing something similar, but in the open you can kind of take shortcuts, which is like we don't need to do this whole pre-training.

We just go right to the model which is best.

Um for SFT.

I think that stage actually looks very similar.

I think we don't have as good of control over which prompts we are using.

I think the closed labs will do a lot of filtering and controlling of the distribution.

We do a lot of mixing, but it's mostly on taking the levels of known data sets and seeing what their performance is on downstream mobile.

for a specific capability and making sure it doesn't degrade other performance.

So if you have like a math eval.

Showing 121–140 of 1,013 · page 7 of 51