Nathan Lambert
speaker
1,013 appearances
1 recordings
1 series
first heard Nov 2024
last heard Nov 2024
Nathan Lambert’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
And this is mostly to be fair to like Quen.
So Quen is a fun example.
example where their 72B model gets a score of like six with Llama 3.1 setting, but a score of 74 with our setting.
So it's like, okay, like we're like competing with Quen instruct in a way.
Like we have to have a setting that is not just what Llama does and I mean all the other labs are doing things like this, which is tailoring their training to a Val.
And it's it's hard to know what they trained on.
We did a lot of work to decontaminate all of our training data sets on the evals that we're doing for development.
We also have an unseen eval suite.
So we do a few methods to check for exact match and overlap on all of the training data sets that we used throughout it, which is some of the final data sets, and we're also gonna release decontaminated versions of like other data sets along the way.
So popular names like um like open.
construct, um NVIDIA's daring anti-ater data set has some contamination on map.
Um for example like the hugging face Numina Math TRR TIR, which is tool integrated reasoning, which was for a math competition, has math contamination.
So we have to remove this.
It's like they were using their model for a Kaggle competition, not for like fair evaluation.
And it really goes to show it's like one, we're releasing the data, but two, we're showing like how easy it is to have contamination.
So it's like
Yes, we need more people to release it.
So we don't know if any of the models we're comparing against trained on test, either on our development set or our unseen set.
It's just like
We can't know.
Showing 301–320 of 1,013 · page 16 of 51
← Previous
Next →