Daniel Kokotajlo

They reward hack.

They are unreliable.

They obviously do cheat and lie.

And the way we've solved it with humans is just checks and balances, decentralization.

You could, like, lie to your boss and keep lying to your boss, but over time it's just not going to work out with you or you become president or something.

Yeah, exactly.

One or the other.

So if you believe in this extremely fast takeoff if a lab is one month ahead, then that's the endgame and this thing takes over.

But even then, I know I'm combining so many different topics.

Even then...

There's been a lot of theories in history which have had this idea of some class is going to get together and unite against the other class.

And in retrospect, whether it's the Marxist, whether it's people who have some gender theory or something, the proletariat will unite or the females will unite or something.

They just tend to...

think that certain agents have shared interests and will act as a result of the shared interest in a way that we don't actually see in the real world.

And in retrospect, it's like, wait, why, why would all the proletariat like, so why think that this lab will have these AIs who are like, there's a million parallel copies and they all unite to secretly, um, uh,

conspire against the rest of human civilization in a way that, even if they are deceitful in some situations.

Okay, so we've been talking about this very much from the perspective of zoom out and what's happening on these log-log plots or whatever.

But 2028 superintelligence, if that happens, what is your sort of... The normal person, what should their reaction to this be?

Sort of, I don't know if emotionally is the right word, but in sort of their expectation of what their life might look like, even in the world where there's no doom.

What do you think of the balance of power idea of slowing down the leading, if there is an intelligence explosion like Dynamic, slowing down the leading companies so that multiple companies are at the frontier?