Eliezer Yudkowsky

There's a, like for some people, like the word intelligence, smartness is not a word of power to them.

It means chess players who, it means like the college university professor, people who aren't very successful in life.

It doesn't mean like charisma, to which my usual thing is like charisma is not generated in the liver rather than the brain.

Charisma is also a cognitive function.

So if you think that smartness doesn't sound very threatening, then superintelligence is not going to sound very threatening either.

It's going to sound like you just pull the off switch.

Like, well, it's superintelligent, but it's stuck in a computer.

We pull the off switch.

Problem solved.

And the other side of it is you have a lot of respect for the notion of intelligence.

You're like, well, yeah, that's what humans have.

That's the human superpower, right?

And it sounds like it could be dangerous, but why would it be?

Have we, as we have grown more intelligent, also grown less kind?

Chimpanzees are, in fact, a bit less kind than humans.

You could argue that out, but often the sort of person who has a deep respect for intelligence is going to be like, well, yes, you can't even have kindness unless you know what that is.

And so they're like, why would it do something as stupid as making paperclips?

Aren't you supposing something that's smart enough to be dangerous, but also stupid enough that it will just make paperclips and never question that?

In some cases, people are like, well, even if you like misspecify the objective function, won't you realize that what you really wanted was X?

Are you supposing something that is like smart enough to be dangerous, but stupid enough that it doesn't understand what the humans really meant when they specified the objective function?