Illia Polosukhin

speaker

1,903 appearances 90 recordings 18 series first heard Feb 2020 last heard 6d ago

Illia Polosukhin’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 51 in all, peaking in Jul 2026 with 8.

newest first · ▶ plays the moment

I guess selling the subscription is that, uh, uh,

So, I mean, obviously I have my hypotheses.

Sure.

And I've kind of expressed some of them.

I mean, like a few years ago, RL was one of them.

And kind of we saw really kind of improvements coming from reinforcement learning.

I definitely think like there is only that much you can stuff, you know, random articles into a model until, you know, it stops learning.

But what we see is if you take some size of the model, like 8 billion parameter, the quality of that model keeps improving.

Meaning like at the same scale of the parameters, we're getting better at how we train them.

And so that to me is the main, like the way I look at these things is like, hey, let's fix the size and see the progression there.

And that progression defines to me the kind of, are we getting better at improving these models?

Yeah, I mean, well, bigger the problem is it's hard to compare apples to apples, right?

Like, hey, is this model better than what it was three months ago?

It is on some metrics, but if you spend 10x more compute, is that actually the improvement we're looking for?

So to me, the kind of...

One of the next improvements is, in general, better training.

And so IRL is kind of part of it, but IRL is still very spotty.

In general, AI is like alchemy.

I don't know if you've read any of the technical papers, but there's like...

We're using learning rate of 0.01 until step 10,000, and then we switch to 0.01, and then at 100 million steps, we're going to anneal it at rate 2x.

Showing 1061–1080 of 1,903 · page 54 of 96