Eiso Kant
speaker
1,158 appearances
1 recordings
1 series
first heard Jul 2026
last heard 23 Jul
Eiso Kant’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jul 2026 with 1.
Appearances
It's the second phase of pre-training with, like, a really dumb way to do a curriculum.
But, like, ultimately what you'd want is a curriculum from token zero to token 30-whatever or 40-train tokens that really truly is the optimal curriculum for the model to learn.
But mid-training is essentially a two-stage curriculum on the web because we do not have to compute...
uh and uh effectively to to try to ablate the perfect curriculum right and so i'm pretty sure that you'll start to see people talking soon about some other terminus tour because now we do this right we talk stage two and stage three and stage four mid training and like but ultimately all we're doing is we're trying to assign a curriculum to the web data that we have to uh to allow the model to learn better
I think at some point as things get compute, as models get cheaper to run, as the next generations of compute, this will become more of a continuous spectrum.
I also think the reason, by the way, you have mid-training and like stage two and stage three is organizational, right?
This is, I think, a thing where that we really try to avoid with the model factory is like...
mid-training exists because there's a mid-training team now right there's people or like people decide to focus on like a mid-training effort but what you really want is engineering and and skill of experiments that allows for a much more continuous spectrum that you don't you have infinite stages now we're not there our compute's not there organization design is not there for it yet
But I think we'll get there.
We'll look back on a couple of years and think, oh my God, it was so cute that we did our pre-training data like this in such a naive way.
Like we barely ordered it.
We didn't really do a good job at like kind of building that curriculum.
We'll get to that in the industry.
We have some ideas.
We're not ready to talk about it yet.
We've been working on them for years.
And I think that's the one thing that's also like, you asked earlier about like, what's not obvious about building a foundation model company is that you are constantly balancing the table stakes work, the recipe, you know, works.
Yeah.
Versus like your, I might break through pure research and, and finding that balance and, and adjusting the percentage to it based on where you are in the race is really important.
And I think, Luke, I think you, it depends when you started as well, right?
Showing 481–500 of 1,158 · page 25 of 58
← Previous
Next →