Max Tegmark

where they proved mathematically that if you just take one goal that you just optimize for, on and on and on indefinitely, that you think is gonna bring you in the right direction.

What basically always happens is, in the beginning, it will make things better for you.

But if you keep going, at some point, it's gonna start making things worse for you again.

And then gradually, it's gonna make it really, really terrible.

So just as a simple, the way I think of the proof is,

Suppose you want to go from here back to Austin, for example, and you're like, okay, yeah, let's just, let's go south, but you put in exactly the right sort of the right direction.

Just optimize that south as possible.

You get closer and closer to Austin.

But there was always some little error.

So you're not going exactly towards Austin, but you get pretty close.

But eventually, you start going away again.

And eventually, you're going to be leaving the solar system.

Yeah.

And they proved it.

It's beautiful mathematical proof.

This happens generally.

And this is very important for AI.

Because even though Stuart Russell has...

written a book and given a lot of talks on why it's a bad idea to have AI just blindly optimize something.

That's what pretty much all our systems do.