Rob Wiblin

That then can drive hyperbolic growth.

Because as you get improvements, then you can run even more.

You can basically expand the effective population of researchers by running more copies of the model.

Have I understood right?

Thank you.

Okay, so how do we tell whether we're talking about a better tool or about more population?

It does seem like there is kind of a fuzzy barrier here, and we're going to go from one to the other, but there's not going to be a sharp cutoff, I imagine.

Okay, so how do we tell if we've just made a better tool or we've made a whole lot of new researchers?

You're saying the proof is in the pudding.

Let's not leave it to the philosophers.

Let's leave it to the people doing benchmarks to figure out whether AI advances are actually speeding up.

A lot of people have the perception, I guess, that AI progress has been slowing down.

You're saying you've tried to create an enormous data set of as many different models over the last four or five years as possible.

This is with Epoch.

who are, this is their wheelhouse, doing this kind of data collection and compilation and aggregation.

So they've tried to collect all these different benchmark scores for many different models going back quite a long way to see is progress speeding up or is it slowing down or is it roughly linear?

And you're saying, at least judged by that measure, the best effort they can do says that it's linear, which is to say, I guess, that we're not making better people, we're making better tools for now.

That's what that would suggest.

Yep, that's right.

So there's all kinds of problems with benchmark scores.