David Duvenaud

Like it's possible that, yes, there's always going to be AIs that have their profitable niche of like making money

human-friendly culture that humans want to consume.

And then the fear is just that we're really riding the tiger and that there's this giant, scary ball of power and optimization that, if it wanted to, could just, I don't know, run over this niche in a million different ways.

Maybe.

I mean, the thing is that when we're that far down the line, it maybe doesn't even matter the most.

Like, I think we're going to see cultural disempowerment happen when as a way of like resting human control.

So if it's the case that humans still have an important veto or like voting rights that matter in some important way, then that's going to create an immense selection pressure to vote.

make some sort of like machines that change their mind or impersonate humans in the sense of like getting to have votes effectively or something like that.

So there's a period where humans still have power.

There's a large selection of pressure to somehow de facto remove that like through cultural means, just because that's one of the means we have to influence or the machines will have to influence humans after that.

then it doesn't really matter what the humans do.

And so, like, it might be that if they've negotiated some, like, small niche that they continue to get, like, there's not much pressure to, like, disempower those humans further.

Right.

So I think this is very unintuitive.

And, you know, maybe there's this midwit meme where it's like intuitively people feel like, OK, someone else getting rich is bad because they're like taking away my stuff.

And then the sort of more enlightened, unintuitive view is like, no, no, actually, you really do want the rich guy to start companies and, you know, make deals and have trade and get to set his own terms in like agreements because it's going to benefit everyone in the long run.

And a rising tide rises all boats.

As for political liberalism, I mean, I feel like all the religious wars in Europe and like throughout the world were so destructive that people eventually realized like, oh, it would be really great if we could all agree to just disagree and not have strong opinions on like, you know, who should rule and what is good.

And who should be killed.

And who should be killed.