Stuart Russell

And this is called the repugnant conclusion.

And, you know, another version is, you know, that we should maximize pleasure.

And that's what we mean by utility policy.

And then you'll get people effectively saying, well, in that case, we might as well just have everyone hooked up to a heroin drip.

And they didn't use those words, but that debate was happening in the 19th century as it is now about AI, that if we get the formula wrong, we're going to have AI systems working towards an outcome that in retrospect would be exactly wrong.

In most cases, not in all, but if we have a wrong political idea, we see it starting to go wrong and we're not completely stupid.

And so we said, okay, maybe that was a mistake.

Let's try something different.

And also we're very slow and inefficient about implementing these things and so on.

So you have to worry when you have corporations or political systems that are extremely efficient.

But when we look at AI systems,

or even just computers in general, right?

They have this different characteristic from ordinary human activity in the past.

So let's say you were a surgeon.

You had some idea about how to do some operation, right?

Well, and let's say you were wrong, right?

That that way of doing the operation would mostly kill the patient.

Well, you'd find out pretty quickly, like after three, maybe three or four tries, right?

But

That isn't true for pharmaceutical companies.