Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
We can go do the technical setup later, but there's no reward model, it's just an RL value function.
Yeah, it's fine.
So that that's a summary of the training.
It's like probably somewhat dense if you haven't heard these terms before, but also informative.
Yeah, so this is for like a general purpose instruction model, which you have to caveat with it.
Because I think if you have different domains, you have much different um compute.
I think uh SFT is something we talked about a lot.
Um
The data site largely kept growing.
We didn't do a lot of sub-sampling results because we're like we're searching for high numbers.
Our final mix is about a million prompts.
Most of them are single-turn.
This model will not be as good at Llama as at multi-turn.
Um we're not a meta AI shop.
We don't need that as much.
It's we don't have the valves for it, but it's about a million prompts at SFT.
If you're using, I can give really specific throughput numbers.
If you're giving using 32 H100s, you can train a 8B model in about a day on R code.
R code is not super optimized because it relies on transformers.
I think you can get about a 40% speed up if you're using kind of really specific code, which is something we might do in the future to only do like Olmo and Lama architectures.
Showing 221–240 of 1,013 · page 12 of 51
← Previous
Next →