David Duvenaud

And as long as humans have some absolute amount of resources, there's going to be this ecological niche to produce culture that these humans want.

And then the point we're just making is that this niche is going to be very, very small in relative terms to this much larger...

sort of more dynamic economy of like political economy, cultural creation of like sort of mostly machine-machine activity.

And that's going to be sort of happening on faster timescales and much larger timescales.

So if, you know, again, like property rights are respected, we manage to keep control of our institutions forever.

Like it's possible that, yes, there's always going to be AIs that have their profitable niche of like making money

human-friendly culture that humans want to consume.

And then the fear is just that we're really riding the tiger and that there's this giant, scary ball of power and optimization that, if it wanted to, could just, I don't know, run over this niche in a million different ways.

Maybe.

I mean, the thing is that when we're that far down the line, it maybe doesn't even matter the most.

Like, I think we're going to see cultural disempowerment happen when as a way of like resting human control.

So if it's the case that humans still have an important veto or like voting rights that matter in some important way, then that's going to create an immense selection pressure to vote.

make some sort of like machines that change their mind or impersonate humans in the sense of like getting to have votes effectively or something like that.

So there's a period where humans still have power.

There's a large selection of pressure to somehow de facto remove that like through cultural means, just because that's one of the means we have to influence or the machines will have to influence humans after that.

then it doesn't really matter what the humans do.

And so, like, it might be that if they've negotiated some, like, small niche that they continue to get, like, there's not much pressure to, like, disempower those humans further.

Right.

So I think this is very unintuitive.

And, you know, maybe there's this midwit meme where it's like intuitively people feel like, OK, someone else getting rich is bad because they're like taking away my stuff.