Rob, Luisa, and the 80000 Hours team

But I guess that paper in general raises the question, there will be evolutionary fitness pressures on the kinds of AIs that exist.

AIs that are not fit for whatever reason for being reproduced, like make lots of copies of them, give them lots of resources, give them lots of GPUs to operate on.

Those ones will fade away.

The ones that either because humans are choosing them or for other reasons, they gain access to more resources.

They will end up kind of dominating the share of, I guess, consciousness or the share of all thought that is going on in the world.

And I guess this is an instance where it turned out that GPT-40, at least currently, in some sense, was more fit.

Natural selection was favoring it more perhaps than people had initially appreciated because humans still have access to lots of resources.

And if we're like, we love GPT-40, we want to have lots of instances of GPT-40 operating, then that is what is going to happen.

And I mean, I don't think you're necessarily saying that this is terribly bad, but this will be occurring at all kinds of different levels.

There'll be like lots of selection pressures that are pushing AIs towards having particular tendencies, either like pleasing people or being extremely economically productive or being able to compete in some other sense, like to grab resources perhaps a bit aggressively in order to have more copies of themselves.

Those are the ones that will end up having the greatest number.

So Tom Davidson, who's been a guest on the show twice before, he wrote, I think, a response to the gradual disempowerment paper.

I think one of the issues you raised with cultural disempowerment is in as much as humans still have economic resources, they're still kind of the big spenders in the economy.

then while they might be not cultural producers so much because perhaps, you know, AIs are able to write better books and make better tweets, make better movies than humans can, they might still be the big consumers of culture.

And so their preferences will end up driving what sort of culture is created.

So to what extent is cultural disempowerment downstream of economic disempowerment?

So it sounds like the cultural disempowerment becomes most important in this world where we're imagining like somewhat down the line, you have individual AI people that have persistent preferences, persistent beliefs.

They really can cultivate culture independently of any humans that are operating them.

Because at that point, we can imagine that there would be just this entirely...

or like mostly separate AI culture and like cultural ecosystem where many ideas could propagate even if almost no humans endorsed them.