Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
So if we're like generating new instructions, then we do SFT on them, it like kind of makes sense that it would make the model worse.
Um but it's like we had these two people doing instruction tuning.
Some people started with like let's compare these DPO alternatives, DPO link nor norm DPO
uh whatever all these things are, figure out what these settings are.
And then some people that are like, where are the open data sets?
Can we get them?
Let's start training on open data sets again.
Where are we falling short of Llama?
So the kind of started as like this algorithmic phase.
And then you get the bearing of like, okay, where are we not doing well with Llama?
Are things just not working?
Let's kill them.
And then it kind of shifts into the second phase of like, let's try to get really specific
specific data.
Um let's kill some of our long tail projects and we're working on specific data and then you continue to mix, you continue to make your valve suite more stable.
When we start the project, the valve suite is not that stable.
It's like, huh, like we're getting crushed by Lama on this thing, but like
the formatting is totally bizarre.
It's like isoteric because we tried to recreate the entire Llama, the valve suite in our code base.
So there's like some evolution there early in the project as you converge on evals, but then it becomes much more exploitative, which is like we build new data sort of specific capabilities.
Showing 601–620 of 1,013 · page 31 of 51
← Previous
Next →