Daniel Kokotajlo

Part of me just wonders like, okay, what if you just need an entirely different kind of data source that's not like pre-training for that, but they have to go out in the real world to get that?

Or maybe they just need to...

it needs to be actively, it needs to be an online learning policy where they need to be actively deployed in the world for them to learn in this way and you show your bottleneck on how fast they can be getting real-world data.

But even in that scenario alone, I can imagine bottlenecks like, oh, you had a benchmark and it got reward hacked for what constitutes AI R&D because you obviously can't have, like, what is the... Maybe you would, but...

is it as good as a human brain?

It's just such an ambiguous thing.

Right now we have benchmarks that get reward hacked, right?

What about the skepticism that...

Look, what you're suggesting with this hyper-efficient hive mind of AI researchers, why no human bureaucracy has just out of the gate worked super efficiently, especially one where they don't have experience working together.

They haven't been trained to work together, at least yet.

And there hasn't been this outer loop RL on like,

We ran a thousand concurrent experiments of different AI bureaucracies doing AI research, and this is the one that actually worked best.

And the analogy I'd use maybe is to humans in the savannah 200,000 years ago.

We know they have a bunch of advantages over the other animals already at this point, but...

Um, the things that make us dominant today, joint stock corporations, state capacities, like this fossil fueled civilization we have, uh, that took so much cultural evolution to figure out.

You couldn't just have like figured it out while in like the Savannah.

I was like, oh, if, you know, if we had built these incentive systems and we, uh, issued dividends, then we could really collaborate here or something.

Why not think that it will take a similar process of huge population growth, huge social experimentation, and upgrading of the technological base of the AI society before they can organize this hypermind collective, which will enable them to do what you imagine intelligence explosion looks like?

I guess the bigger the organization, even if everybody is aligned, I think some of your responses addressed whether they will be aligned on goals.

I mean, you did address the whole thing, but I will just point this out.