Eliezer Yudkowsky

Why?

Because I said tiny molecular paperclips, this got then mutated to paperclips, this then mutated to, and the AI was in a paperclip factory.

So the original story is about how

You lose control of the system.

It doesn't want what you tried to make it want.

The thing that it ends up wanting most is a thing that even from a very embracing cosmopolitan perspective, we think of as having no value.

And that's how the value of the future gets destroyed.

Then that got changed to a fable of like, well, you made a paperclip factory and it did exactly what you wanted, but you asked it to do the wrong thing, which is a completely different failure mode.

So that's more than Brave New World.

Yeah.

If you can solve the problem of making something want exactly what you want it to want, then you get to deal with the problem of wanting the right thing.

But first you have to solve the alignment.

First you have to solve inner alignment.

Inner alignment.

Then you get to solve outer alignment.

Like first you need to be able to point the insides of the thing in a direction, and then you get to deal with whether that direction expressed in reality is like the thing that aligned with the thing that you want.

Are you scared?

Of this whole thing?

Probably.

I don't really know.