David Duvenaud

So if we did take our hands off the reins, we might expect that the future AI civilization would create their own sub-AI slaves or something that were treated poorly.

And so if the idea is that, no, no, it's fine, they'll be enlightened enough not to do that,

then maybe you're okay.

But I feel like it's weird to think that they'll somehow recognize this as like this really important moral truth that we don't see.

Yeah, I mean, the question, though, is, like, what sort of dynamics are you setting up, right?

If you just say, I'm siding with the AIs, but they themselves are still in this, like, competitive race, then you still are sort of, like, taking your hands off the wheel and saying, okay, well, their culture and morality is still going to evolve, and where do we think that's going to lead us is sort of the important question then.

So I agree with it.

Okay, I guess I'll say that's conflating two positions, one of which I agree with and one I don't, which is I think almost by definition, we want human values to rule forever.

And if you're saying like, no, no, no, I want some progress or evolution to happen, it's like, great, that is now the value that you want to block in forever, even if it involves lots of object-level change.

As for whether humans are actually in the loop making decisions, I think it's probably the case that whatever you want to optimize eventually does look like this is mostly handed off.

And then the caveat is that sort of on reflection, if the AI has ever cared to ask the humans, like, how do you agree with how we've been running things?

That the humans would be like, two thumbs up.

That's what I would have done.

Like, keep up the good work.

Yes.

So I think that's a huge problem, but it's also a huge problem for anyone who has a proposal about what to do with the future.

Right.

So it's not like this particularly affects my claim about how we should run things.

Yes, exactly.

Okay.