Daniel Kokotajlo

You have a 5x at some point, and then...

Dot, dot, dot, you have 1,000x AI progress within the matter of a year.

Maybe that's the part I'm like, wait, how did that happen exactly?

Yeah.

So what's the story there?

So maybe the big picture thing I have with the intelligence explosion is we can go through the specific arguments about how much will the automated coder be able to do and how much will the superhuman AI coder be able to do.

But on priors, it's just like such a wild thing to expect.

And so...

Before we get into all the specific arguments, maybe you can just address this idea that, like, why not just start off with, like, 0.01% chance this thing might happen, and then you need extremely, extremely strong evidence that it will before making that your moral view.

I mean, you are saying that this transition, these previous transitions have been smoother than the one you're anticipating, right?

You guys are the conservative crowd, you know.

Okay, now that we've brought up the intelligence explosion, let's just discuss that because I'm kind of skeptical.

It doesn't really seem to me that a notable bottleneck to AI progress or the main bottleneck to AI progress is the amount of researchers, engineers who are doing this kind of research.

It seems more like compute or some other thing is a bottleneck.

And the piece of evidence is that when I talk to my AI researcher friends,

At the labs, they say there's maybe 20 to 30 people on the core pre-training team that's discovering all these algorithmic breakthroughs.

If the headcount here was so valuable, you would think that, for example, Google DeepMind would take not just everybody from all their smartest people, not just from DeepMind, but for all of Google and just put them on pre-training or RL or whatever the big bottleneck was.

You'd think OpenAI would hire...

every single Harvard math PhD, and in six months, you're all going to be trained up on how to do AI research.

They don't seem that... I mean, I know they're increasing headcount, but they don't seem to treat this as the kind of bottleneck that...