Emad Mostaque
👤 SpeakerAppearances Over Time
Podcast Appearances
You can just teach it a good base, and then it goes from there.
And it scores higher on a human evaluation and other metrics, but we don't know what the right data set is.
It's just right now, we said, let's scale.
More data, more compute.
Now we're like, what's the right data?
What's the right compute?
Like our image model, we have over 120 different clusters of images.
Only like nine are used, like 95% of the time.
All the rest of the data is just bunkum.
What does that look like for a language model?
Like, do you need to train it on all of those auto-generated transcripts of, like, Spider-Man pulling out someone's tooth on YouTube and all these weird videos?
There's a whole subculture of generated videos where you have, like, Spider-Man and SpongeBob SquarePants and Mickey Mouse, like, having a fight and stuff like that.
I got to find these corners.
It's a deep, dark area of YouTube.
You don't want to go there, man.
Yes, or like a group of humans coming together, there's suddenly a race condition where it just goes.
It's not trying to do something bad.
The humans don't want to do something bad, but it happens.
Just like the example I always give is YouTube optimized for engagement, which then optimized for extreme content, which is optimized for ISIS.
Nobody in YouTube wanted ISIS to do well.