Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
We found a lot of contamination out there, but we don't know what other people are doing and it's kind of
I think there's probably a snarky paragraph in the paper which is like, We don't know which of any of these models trained on the EFLs.
It's like
Okay, where's for the best?
Um techniques like that.
We're looking at data sets.
Um people are working on model diagnostic, but we're looking at data sets.
And there was this weird thing.
We'll we'll talk about data sets where essentially the biggest thing we're doing is exact prompt match.
So if a certain data set training data set has more than like a two percent exact prompt match with one of our test sets, which is like that's the percentage of the test set in the prompts exactly, we consider that contaminated and remove them.
The other thing is like can you detect if models trained on certain test sets?
So I
Earlier in this year, my big project was reward bench, which is like trying to build an ecosystem for evaluating reward models.
Um
Now it is going, there's lots of academic papers on it.
But one of the weirdest things we found is recently substantial contamination on
Reward bench prompts, which was taken mostly from other test sets, was generated by Llama Instruct with the Magpie method.
So MagPy is a synthetic data method that manipulates the chat template to get the model to generate prompts in distribution of what it was trained on.
So this is the type of like weird head scratching you need to do to do like, is a model trained on something?
You can't prove it, but it
Showing 321–340 of 1,013 · page 17 of 51
← Previous
Next →