David Duvenaud

Maybe I'm not bothered by that.

But then I think, wait, wait, wait.

So like right now I have kids and they're going to be having kids.

And if a thousand years from now, some other race is taking over, where's the day where my kids get like killed or starved or like replaced or uploaded or whatever?

Right.

And it's one of these things where it's just I think it's a skill, basically having coherent preferences about the future.

And the more people spend time actually thinking about it, I think the more they would say like, oh, wait, I actually don't want to take my hands off the wheel.

The default competitive future probably is missing a whole bunch of stuff that I care about.

Actually, I would push back.

I would say the gradual disempowerment paper doesn't really talk about human versus AI control.

It talks about the alignment of our current institutions and how they're going to become less aligned once AI are more in the loop and taking over from humans.

I mean, I guess to me, again, the question is not like AIs versus humans.

It's more like, what are the idiosyncratic things that humans value that we don't expect to be competitive in the long run versus the sort of values of unfettered competition and natural selection?

Like some things that we value, like communication and sites and memory, like we don't have to fight for those.

Like whatever future beings are around are probably going to be able to see and talk and remember that.

But idiosyncratic things like children's laughter or the family farm or the particular languages we speak or something are things that will be out-competed and replaced if we don't preserve them.

So to the extent that any of us value anything that isn't going to stand the test of time, we sort of have to accept that we're going to lose those or try to somehow do this crazy task of aligning our whole civilization to protect these idiosyncratic, non-competitive values.

Sure.

So one thing is just that I think human politicians will gradually let themselves be more puppeted or like pass-throughs for things like ChatGPT.

And this isn't necessarily a bad thing in the short run.