Max Tegmark

We have something called the loss function that we're just minimizing or reward function we're just maximizing.

And capitalism is exactly like that too.

We wanted to get stuff done more efficiently than people wanted.

So we introduced the free market.

Things got done much more efficiently than they did in, say, communism, right?

And it got better.

But then it just kept optimizing and kept optimizing.

And you got ever bigger companies and ever more efficient information processing and now also very much powered by IT.

And...

Eventually, a lot of people are beginning to feel, wait, we're kind of optimizing a bit too much.

Why did we just chop down half the rainforest?

And why did suddenly these regulators get captured by lobbyists and so on?

It's just the same optimization that's been running for too long.

If you have an AI that actually has power over the world and you just give it one goal and just keep optimizing that, most likely everybody's going to be like, yay, this is great in the beginning.

Things are getting better.

But it's almost impossible to give it exactly the right direction to optimize in.

And then eventually all hay breaks loose, right?

Nick Bostrom and others have given examples that sound quite silly.

Like, what if you just want to tell it to cure cancer or something, and that's all you tell it?

Maybe it's going to decide to...