Daniel Kokotajlo

Yeah.

So we were just getting lunch with a prominent Washington, D.C.

political journalist, and he was making the point that when he talks to these Congress people, when he talks to political leaders, none of them are at all awake to the possibility even of stronger AI systems, let alone HEI, let alone superhuman intelligence.

I think a lot of your...

forecast relies on at some point, not only the US president, but also Xi Jinping, wake up to the possibility of a super intelligence and the stakes involved there.

Why think that even when you show Trump the remote worker demo, he's going to be like, oh, and therefore in 2028, there will be a super intelligence.

Whoever controls that will be God Emperor forever.

Maybe not that extreme.

You see what I'm saying?

Like, why not?

Why wouldn't he just be like, oh, there'll be a stronger remote worker in 2029, a better remote worker in 2031.

Yeah.

Is this good or bad?

That the president and the companies are like... I think it's bad.

So if the big bottleneck to the good future here is just putting in not this Eliezer-type galaxy brain, high volatility, you know, there's a 1% chance this works, but we've got to come up with this crazy scheme in order to make alignment work.

But rather, as Daniel, you were saying, more like, hey, do the obvious thing of making sure you can read how the AI is thinking.

Make sure you're monitoring the AIs.

Make sure they're not forming some sort of hive mind where you can't really understand how the millions of them are coordinating with each other.

To the extent that, and I want to stay short forward, but to the extent that it is a matter of prioritizing it, closing all the obvious loopholes, it does make sense to leave it in the hands of people who have at least said that this is a thing that's worth doing, have been thinking about it for a while.

And I worry about...