David Duvenaud

So we don't really have to worry about that scenario.

But the scenario where they're just kind of like this gray goo that we think is a big waste, that's what we need to avoid.

And if you and I are on Earth flourishing for a long time, and the state and all our civilization apparatus is acting in our interest, and we decide, hey, actually, it would be amazing to create this type of future, then just as part of serving our interests, we would end up creating that amazing future.

But that doesn't mean that I don't take a lot of morality very seriously.

I'm just trying to say the sort of fact of the matter is determined by what's in our heads and then also whatever conditions that imposes on the world being good.

Well, and I think it's a really weird corner case to imagine this world where like we die, but then our desires are ultimately fulfilled.

Like that just seems like, yes, in principle it could happen, but it would be like this weird corner case because probably if we die, something else has gone horribly wrong.

Right.

I mean, I think that just puts a cap on sort of like what's the best we can hope for in terms of satisfying everyone's preferences, but that we were already sort of in that situation.

So I guess I'll say, given that we already have to have some sort of like compromise or not everyone's going to get what they want, we should at least work to make it so that at least some people or some compromise of humans gets what it wants as opposed to just competition gets what it wants.

Sure.

Well, at least first I'll say the most common arguments I hear in favor of this.

So one is...

pretty soon we're going to have these amazing AIs.

So they're going to handle this for us.

And we don't really need to worry about these kind of coordination problems.

And I guess I feel like, yes, if there was a big jump in capabilities and everybody got it on the same day and everybody asked their AI, what should I do for the good of humanity and did that, then that would be a recipe for really good outcomes.

But I don't think that's what was going to happen.

We're going to continue to see people gradually getting more powerful AIs and those gradually getting spread out according to roughly to power level.

And people continuing to optimize mostly for their own interests just due to competitive pressures.