David Duvenaud

And I think

They will be very impressive.

And if they want to, like, be sympathetic or like sort of moral patience from our point of view, they'll be able to make themselves into that.

So I really don't expect it to look like there's this alien mass of activity happening, but more like the Chagas with the human face.

But in a sort of more sort of genuine way, I think they will be able to say like, oh, no, I actually do spend some like fraction of my computes.

thinking about the world in a way that you think is valuable or something like that.

So that question of how sort of morally valuable this giant machine economy shoggoth is kind of dominates that question of what positive sub-dynamics still exist.

Totally.

And I think you're getting at a big open empirical question, which is sort of like, what is stable at the top?

Like, should we expect there to be some big like global government that forms and then consolidates power and like lasts forever?

Or is it going to be the case that just like the Roman Empire, like it became more and more powerful, but then there was like new religions that formed inside and there was sort of this like

cultural competitions, and all these reasons why every time an empire falls, it eventually ossifies and some sort of internal competition makes it fall.

Because if the case is that we're all heading towards some global government and it's going to be really hard to change, then we need to be investing in ways that we can, like,

steer this or slow it down or make room for pluralism or something like that.

But if we think that the default at the scales we care about is just that it's hard to coordinate, there's always going to be runaway competition eating any surplus, then we'll want to invest in like, no, no, how do we actually slow that process down and make sure that we can preserve these non-competitive values?

Sure, yeah.

And one thing I want to say is, I mentioned a fixed pie, but the pie itself might be growing.

And it might be growing exponentially as the solar system is colonized.

But the point is that it's zero-sum in the sense that humans don't contribute to the pie growing faster or smaller in the way that economic activity does make the pie grow today.

So it might not be a fixed pie, but it's still the case that it's zero-sum when you compete with the other people for your share.