David Duvenaud

And it's one of these things where it's just I think it's a skill, basically having coherent preferences about the future.

And the more people spend time actually thinking about it, I think the more they would say like, oh, wait, I actually don't want to take my hands off the wheel.

The default competitive future probably is missing a whole bunch of stuff that I care about.

Actually, I would push back.

I would say the gradual disempowerment paper doesn't really talk about human versus AI control.

It talks about the alignment of our current institutions and how they're going to become less aligned once AI are more in the loop and taking over from humans.

I mean, I guess to me, again, the question is not like AIs versus humans.

It's more like, what are the idiosyncratic things that humans value that we don't expect to be competitive in the long run versus the sort of values of unfettered competition and natural selection?

Like some things that we value, like communication and sites and memory, like we don't have to fight for those.

Like whatever future beings are around are probably going to be able to see and talk and remember that.

But idiosyncratic things like children's laughter or the family farm or the particular languages we speak or something are things that will be out-competed and replaced if we don't preserve them.

So to the extent that any of us value anything that isn't going to stand the test of time, we sort of have to accept that we're going to lose those or try to somehow do this crazy task of aligning our whole civilization to protect these idiosyncratic, non-competitive values.

Sure.

So one thing is just that I think human politicians will gradually let themselves be more puppeted or like pass-throughs for things like ChatGPT.

And this isn't necessarily a bad thing in the short run.

I mean, good politicians already rely heavily on human advisors.

And I think machine advisors are going to be able to make our political decisions

parties and just representation mechanisms work better in a lot of ways.

So the politicians that use AIs just for normal everyday business are going to be more effective and we're going to feel like they represent our interests better in the short run at least.

The other big thing that's going to be going on is I think people are going to be afraid of losing their jobs.