Konstantin Kisin

I don't know if this is the main point, but...

You and I have a survival instinct, and I think we're productive members of society.

Sure.

And we've built a society where people who are selfish, who are trying to make their own lives better, as a natural byproduct of their actions, also make society better.

And, you know, this is a...

people observe, like, you take somebody like Mark Zuckerberg or Elon Musk or whatever, and in another era, they might have been Napoleons and Alexanders just going through the fields and... I don't think they'd actually be mass-working people, but, like, there's a certain kind of person who would just go around just, um...

causing huge amounts of destruction.

But in today's world, there's capitalism and industry and whatever.

They can use those energies to do incredibly productive things, like build rockets that go to space.

So yeah, similarly, I don't think it's necessary that AI never has its own objectives or its own sense of morality.

I think there's ways to build this.

And in fact, it's necessary to build civilization in a way that's compatible with self-interested actors.

And the reason is because even if today we have AIs that we can program to only care about whatever it means to care about humans, someday somebody's going to build an AI that's self-interested.

And if...

there's going to be some selection between AIs.

Which is the AI that survives?

All these AIs are let loose.

Which one gathers resources becomes more powerful?

One of the reasons that an AI will be successful is that it actually cares about being successful.

We need a civilization that's robust to AIs that partly also care about themselves.