Stuart Russell

So he's incredibly greedy.

You know, we call this the Midas touch.

And we think of the Midas touch as being like, you know, that's a good thing, right?

Wouldn't that be cool?

But what happens?

So he, you know, he goes to drink some water and he finds that the water has turned to gold.

And he goes to eat an apple and the apple turns to gold.

And he goes to, you know, comfort his daughter and his daughter turns to gold.

And so he dies in misery and starvation.

So this applies to our current situation in two ways, actually.

So one is that I think greed is driving us to pursue a technology that will end up consuming us.

And we will perhaps die in misery and starvation instead.

What it shows is how difficult it is to correctly articulate what you want the future to be like.

For a long time, the way we built AI systems was we created these algorithms where we could specify the objective, and then the machine would figure out how to achieve the objective and then achieve it.

So we specify what it means to win at chess or to win at Go, and the algorithm figures out how to do it, and it does it really well.

So that was standard AI up until recently.

And it suffers from this drawback that, sure, we know how to specify the objective in chess, but how do you specify the objective in life?

What do we want the future to be like?

Well, really hard to say.

And almost any attempt to write it down precisely enough for the machine to bring it about would be wrong.