Max Tegmark

So if anyone goes over, they'll start dragging the others down too.

And so ultimately it's in the selfish...

interests also of the people in the companies to slow down when you start seeing the contours of the cliff there in front of you, right?

And the problem is that even though the people who are building the technology

And the CEOs, they really get it.

The shareholders and these other market forces, they are people who don't honestly understand that the cliff is there.

They usually don't.

You have to get quite into the weeds to really appreciate how powerful this is and how fast.

And a lot of people are even still stuck again in this idea that...

in this carbon chauvinism, as I like to call it, that you can only have our level of intelligence in humans, that there's something magical about it.

Whereas the people in the tech companies...

who build this stuff, they all realize that intelligence is information processing of a certain kind.

And it really doesn't matter at all whether the information is processed by carbon atoms in neurons and brains or by silicon atoms in some technology we build.

So you brought up capitalism earlier, and there are a lot of people who love capitalism and a lot of people who really, really don't.

It struck me recently that what's happening with capitalism here is exactly analogous to the way in which superintelligence might wipe us out.

You know, I studied economics for my undergrad, Stockholm School of Economics.

Yay.

No, I tell them.

So I was very interested in how you could use market forces to just get stuff done more efficiently, but give the right incentives to the market so that it wouldn't do really bad things.

So Dylan Hadfield-Manel, who's a professor and colleague of mine at MIT, wrote this really interesting paper with some collaborators recently.