Max Tegmark

But if you build an artificial general intelligence, the mind space that you can design is much, much larger than just a specific subset of minds that can evolve.

So an AGI mind doesn't necessarily have to have any self-preservation instinct

It also doesn't necessarily have to be so individualistic as us.

Like imagine if you could just... First of all, we're also very afraid of death.

I suppose you could back yourself up every five minutes and then your airplane is about to crash.

You're like, shucks, I'm just...

I'm going to lose the last five minutes of experiences since my last cloud backup.

Dang!

It's not as big a deal.

Or if we could just copy experiences between our minds easily, which we could easily do if we were silicon-based, right?

Then maybe we would feel a little bit more like a hive mind, actually.

So I don't think we should take for granted at all that AGI will have to have any of those sort of

competitive as alpha male instincts.

On the other hand, this is really interesting because I think some people go too far and say, of course, we don't have to have any concerns either that advanced AI will have those instincts because we can build anything we want.

That there's a very nice set of arguments going back to Steve Omohundro and Nick Bostrom and others just pointing out that when we build machines,

we normally build them with some kind of goal, you know, win this chess game, drive this car safely or whatever.

And as soon as you put in a goal into machine, especially if it's kind of open-ended goal and the machine is very intelligent, it'll break that down into a bunch of sub goals.

And one of those goals will almost always be self-preservation because if it breaks or dies in the process, it's not going to accomplish the goal, right?

Like suppose you just build a little, you have a little robot and you tell it to go down,

the star market here and get you some food, make you cook an Italian dinner, you know.