Scott Alexander

There are a bunch of ways in which we could be underestimating, and there are a bunch of ways in which we could be overestimating.

And we're going to hopefully continue to think more about this afterwards and continue to iteratively refine our models and come up with better guesses and so forth.

Great question.

We still have to, like, stumble through all the discoveries that we've made since 2017.

We still have to, like...

figure out that language models are a thing.

We still have to figure out that you can fine-tune them with RL.

So all those things would still have to happen.

How much faster would they happen?

Maybe 5x faster because a lot of the small-scale experiments that these people do in order to test out ideas really quickly before they do the big training runs would happen much faster because they're just like liquidly being spit out.

I'm not very confident in that 5x number.

It could be lower, it could be higher, but that was sort of like roughly what we were guessing.

Our 5x, by the way, is for the algorithmic progress part, not for the overall thing.

So in this hypothetical, according to me, basically things would be going like 2.5x faster, where the algorithms would be advancing at 5x speed, but the compute is still stuck at the usual speed.

The way that we did our takeoff forecast was basically by breaking down how we think the intelligence explosion would go into a series of milestones.

First, you automate the coding.

Then you automate the whole research process, but in a very similar way to how humans do it with, like,

you know, teams of agents that are about human level, then you get to superhuman level and so forth.

And so we broke it down as these milestones, you know, the superhuman coder, superhuman AI researcher, and then super intelligent AI researcher.

And the way we did our forecast was we basically, well, for each of these milestones, we were like, what is it going to take to make an AI that has, that achieves that milestone?