Dwarkesh

So you're training them on two different things.

First, you're rewarding them for this deceptive behavior.

Second of all, you're punishing them.

And we don't have a...

great prediction for exactly how this is going to end.

One way it could end is you have an AI that is kind of the equivalent of the startup founder who really wants their company to succeed, really likes making money, really likes the thrill of successful tasks.

They're also being regulated and they're like, yeah, I guess I'll follow the regulation.

I don't want to go to jail.

But it's not like robustly, deeply aligned to, yes, I love regulations.

My deepest drive is to follow all of the regulations in my industry.

So we think that an AI like that, as time goes on and as this recursive self-improvement process goes on, will kind of get worse rather than better.

It will move from kind of this vague superposition of, well, I want to succeed.

I also want to follow things to like being smart enough to genuinely understand its goal system and being like, my goal is success.

I have to pretend to want to do all of these moral things while the humans are watching me.

That's what happens in our story.

And then at the very end, the AIs reach a point where the humans are pushing them to have clearer and better goals because that's what makes the AIs more effective.

And they eventually clarify their goals so much that they just say, yes, we want task success.

We're going to pretend to do all these things well while the humans are watching us.

And then they grow, they outgrow the humans, and then there's disaster.

Yeah, we don't know how this will work at the limit of all these different training methods, but we're also not completely making this up.