David Duvenaud

Yeah, yeah.

So I totally agree that I think...

It's easy to make the case that in the long run, some sort of global coordination is stable.

Once you have a lot of agency and coordination, you can use it to get more.

The question is, will our values survive the chaos in the generation of that global power or whatever it is?

Basically, there's this window where right now humans have a lot of influence, our institutions and cultures kind of serve us, but we don't have very good coordination ability.

then at some point we're going to have like diminishing influence and diminishing sort of like cultural cache and all those things at the same time as our ability to coordinate is going to go up.

So the question is sort of like, what's going to happen first?

Are we going to be all competed and then later the machines coordinate or do we get to be part of that like global coordination process?

Yeah, exactly.

That's exactly what I have in mind when I'm saying that our ability to coordinate will increase.

And then the question is sort of, will human influence survive long enough to strongly influence that process?

Yeah, so the sort of only thing we can do is to introduce a new fitness landscape where sort of who the government deserves worthy of UBI is the new like fitness function that is going to be just optimized by natural selection, right?

Yeah, absolutely.

So if we just allow unfettered reproduction like we do today, then it's unstable in the long run.

I mean, the problem is it's kind of like, well, what do you even want?

And I think sort of like the best we can hope for is that we do live under such a regime where we've thought hard about what is the sort of...

fitness function or like how we redistribute wealth or something.

And the fear is that we don't think it through very well.

And we end up locked in this like new Malthusian condition where everyone has to like have as many kids as possible or something like that, um, to, to like maximize their share, to maintain their share of wealth or something like that.