David Duvenaud

So yeah, my fear is that the business as usual doesn't give us such jumping capabilities that we're suddenly able to coordinate in a way we weren't before.

The other common argument is, well, if people a thousand years ago got their way, we wouldn't have made all the moral progress that we've made since then.

And so it would have been a huge mistake from our point of view today to have let them lock in.

So by induction, it will be a huge mistake from the point of view of future beings to have let us lock in.

And I think that's kind of like a moral optical illusion in that, yes, if you measure sort of

like moral progress by like difference from our current moral standards, then if you go backwards in time, the further back you go, sort of the worst things get sort of monotonically.

And so if you just extrapolate that, it looks like, oh, so if we go forward in time and we continue allowing this sort of moral progress to happen, things are going to get better.

But I think what's actually happening is we're just measuring difference from our current values.

So

we should actually see this lion have a kink exactly at the present day and things just to continue to, however they evolve, however moral standards evolve, to look worse and alien and wrong from our current point of view.

So I think from the point of view of future beings, whoever is alive is going to be glad that the past wasn't able to lock in their different values, but they're basically going to be happy that they're in power.

So in a sense, we don't have to worry about the future beings.

They're kind of one no matter what.

The only thing that remains to worry about is locking in our current values in some sense.

And of course, we probably don't want to lock in all these short-term, maybe irrelevant details or local adaptations.

We probably want to lock them in at some point

larger sort of more abstract like big-minded view but i think if we do by definition want value lock-in because if you're okay with your value changing it's not really clear in what sense it's actually a value of yours

I totally agree.

But I guess in terms of what we should do, I think it doesn't end up mattering because even if we're moral realists, the question is, do the AI successors that we build care about that morality?

And I think the default is, no, they care about growth just in the way that animals don't care about morality and evolution doesn't care about morality.