Pete Huang

The question is this, what happens if the trends continue like they have been?

If we take the same amount of improvement that went from GPT-2 to GPT-4, again from that preschooler level to the high schooler level, and then just start at GPT-4, where do we end up?

I mean, this is kind of a dumb comparison, but preschool to high school is like 13, 14 years of education for a human.

If you give a high schooler another 13, 14 years of education, they would have a PhD.

And because this is all happening in a computer, when you have one AI model that is as good as a human with a PhD, you actually have millions, hundreds of millions, billions of them.

So if you suddenly had a million PhD level AI models that could just do more AI research, doesn't that pretty much guarantee that you will have ASI very shortly afterwards?

Okay, let's take a breather for a second.

We were just talking about GPT-2 and how it was no smarter than a preschooler just a few moments ago, and now we're talking about Skynet?

Like, geez, that's a bit of a jump, isn't it?

The specific arguments behind it kind of go like this.

If you want to say, let's take the same improvement from GPT-2 to GPT-4 and see what happens if we continue,

then you first have to know how much actual improvement there was in GPT-2 to GPT-4, right?

Leopold does this by measuring how much computing went into training these models.

Again, remember that GPT-2 to GPT-4 was simply a matter of more training with the same general model architecture.

Leopold breaks it down into three buckets.

Bucket number one is just how big the computers are that you use to train.

Bigger and more efficient computers are more powerful and therefore you put more training in these models.

Big machine is better, basically.

The second bucket is how you train those models, the algorithms that you run, the methods that you use.

For example, you could train a model in a different way such that the whole thing runs faster and is more efficient.