Stuart Russell

Okay, so if you accept that it's possible, and if you accept that it's probably going to happen, the point that you're making that, how does it go wrong?

a valid question.

Without that, without an answer to that question, then you're stuck with what I call the gorilla problem, which is the problem that the gorillas face, right?

They made something more intelligent than them, namely us, a few million years ago, and now they're in deep doo-doo.

So there's really nothing they can do.

They've lost the control.

They failed to solve the control problem of controlling humans, and so they've lost control.

So we don't want to be in that situation.

And if the gorilla problem is the only formulation you have, there's not a lot you can do, right?

Other than to say, okay, we should try to stop.

We should just not make the humans or in this case, not make the AI.

And I think that's really hard to do.

I'm not actually proposing that that's a feasible course of action.

I also think that if properly controlled AI could be incredibly beneficial.

But it seems to me that there's a

There's a consensus that one of the major failure modes is this loss of control, that we create AI systems that are pursuing incorrect objectives.

And because the AI system believes it knows what the objective is, it has no incentive to listen to us anymore, so to speak, right?

It's just carrying out the strategy that it has computed as being the optimal solution.

And it may be that in the process, it needs to acquire more resources to increase the possibility of success or prevent various failure modes by defending itself against interference.

And so that collection of problems, I think, is something we can address.