Eliezer Yudkowsky

Asking you how you would take over an alien world where you can do a whole lot of cognition at John von Neumann's level, as many of you as it takes, and the aliens are moving very slowly.

I think to understand the full depth of the problem, we actually, I do not think it is possible to understand the full depth of the problem that we are inside without

understanding the the problem of facing something that's actually smarter not a malfunctioning recommendation system not something that isn't fundamentally smarter than you but is like trying to steer you in a direction yet no like if we if we solve the the weak stuff this the if we solve the weak ass problems the strong problems will still kill us is the thing and i think that to understand the situation that we're in you want to like tackle the conceptually difficult part and

I have been struggling for years to convey this intuition.

The most success I've had so far is, well, imagine that the humans are running at very high speeds compared to very slow aliens.

Because people understand the power gap of time.

They understand that today we have technology that was not around 1,000 years ago, and that this is a big power gap, and that it is bigger than... Okay, so what does smart mean?

When you ask somebody to imagine something that's more intelligent, what does that word mean to them, given the cultural associations that that person brings to that word?

For a lot of people, they will think of like, well, it sounds like a super chess player that went to double college.

And because we're talking about the definitions of words here, that doesn't necessarily mean that they're wrong.

It means that the word is not communicating what I want it to communicate.

The thing I want to communicate is the sort of difference that separates humans from chimpanzees.

But that gap is so large that you ask people to be like, well, human, chimpanzee, go another step along that interval of around the same length, and people's minds just go blank.

Like, how do you even do that?

And I can try to break it down and consider what it would mean to send a schematic for an air conditioner 1,000 years back in time.

Now, I think that there is a sense in which you could redefine the word magic to refer to this sort of thing.

And what do I mean by this new technical definition of the word magic?

I mean that if you send a schematic for the air conditioner back in time, they can see exactly what you're telling them to do.

But having built this thing, they do not understand how it output cold air.

Because the air conditioner design uses the relation between temperature and pressure.