Daniel Kokotajlo

They haven't been trained to work together, at least yet.

And there hasn't been this outer loop RL on like,

We ran a thousand concurrent experiments of different AI bureaucracies doing AI research, and this is the one that actually worked best.

And the analogy I'd use maybe is to humans in the savannah 200,000 years ago.

We know they have a bunch of advantages over the other animals already at this point, but...

Um, the things that make us dominant today, joint stock corporations, state capacities, like this fossil fueled civilization we have, uh, that took so much cultural evolution to figure out.

You couldn't just have like figured it out while in like the Savannah.

I was like, oh, if, you know, if we had built these incentive systems and we, uh, issued dividends, then we could really collaborate here or something.

Why not think that it will take a similar process of huge population growth, huge social experimentation, and upgrading of the technological base of the AI society before they can organize this hypermind collective, which will enable them to do what you imagine intelligence explosion looks like?

I guess the bigger the organization, even if everybody is aligned, I think some of your responses addressed whether they will be aligned on goals.

I mean, you did address the whole thing, but I will just point this out.

That is not the part I'm skeptical of.

I am more skeptical of just like,

even if you're all aligned and want to work together, do you fundamentally understand how to run this huge organization?

And you're doing it in ways that no human has had to before.

You're getting copied incessantly.

You're running extremely fast.

You know what I'm saying?

And so it's a complicated thing, and I'm just not sure why you think we build this bureaucracy, or the AIs build this bureaucracy,

Five years?