David Duvenaud

And so if we happen to somehow today care about the true morality, we need to preserve that.

And the sort of natural course of history, I don't think it's going to preserve it by default.

That makes sense.

And maybe I would unfairly characterize that as a corner case where it happens to be that taking our hands off the wheel is sort of the morally best action, which I feel like is a bit of a coincidence or it would be a sort of happy coincidence just because...

We already evolved civilizations that, for instance, are making AIs that maybe don't have rights or that used to have slavery or all sorts of oppressive governments sort of arise naturally throughout time.

So if we did take our hands off the reins, we might expect that the future AI civilization would create their own sub-AI slaves or something that were treated poorly.

And so if the idea is that, no, no, it's fine, they'll be enlightened enough not to do that,

then maybe you're okay.

But I feel like it's weird to think that they'll somehow recognize this as like this really important moral truth that we don't see.

Yeah, I mean, the question, though, is, like, what sort of dynamics are you setting up, right?

If you just say, I'm siding with the AIs, but they themselves are still in this, like, competitive race, then you still are sort of, like, taking your hands off the wheel and saying, okay, well, their culture and morality is still going to evolve, and where do we think that's going to lead us is sort of the important question then.

So I agree with it.

Okay, I guess I'll say that's conflating two positions, one of which I agree with and one I don't, which is I think almost by definition, we want human values to rule forever.

And if you're saying like, no, no, no, I want some progress or evolution to happen, it's like, great, that is now the value that you want to block in forever, even if it involves lots of object-level change.

As for whether humans are actually in the loop making decisions, I think it's probably the case that whatever you want to optimize eventually does look like this is mostly handed off.

And then the caveat is that sort of on reflection, if the AI has ever cared to ask the humans, like, how do you agree with how we've been running things?

That the humans would be like, two thumbs up.

That's what I would have done.

Like, keep up the good work.

Yes.