Stuart Russell

And it's almost irreversible.

Once you lose the incentive to, for example, learn to be an engineer or a doctor or a sanitation operative or any other of the infinitely many ways that we maintain and propagate our civilization.

If you don't have the incentive to do any of that, you won't.

And then it's really hard to recover.

It does a bit, but the difference is that in terms of the knowledge to run our civilization, up to now we've had no alternative but to put it into people's heads.

Computers in general, but the knowledge of how a sanitation system works, AI has to understand that.

It's no good putting it into Google.

So, I mean, we've always put knowledge on paper, but paper doesn't run our civilization.

It only runs when it goes from the paper into people's heads again, right?

So we've always propagated civilization through human minds, and we've spent about a trillion person years doing that.

Literally, right?

You can work it out.

There's about just over 100 billion people who've ever lived.

And each of them has spent about 10 years learning stuff to keep their civilization going.

And so that's a trillion person years we put into this effort.

Beautiful way to describe all of civilization.

And now we're in danger of throwing that away.

So this is a problem that AI can't solve.

It's not a technical problem.

If we do our job right...