David Duvenaud

thinking about this kind of institution design, A, with LLM as this new tool that we can use to help do better job, and B, with this sort of more radical futurism approach and saying like, the stakes are high, and it's not just a question of like, do we get better outcomes on the margin?

It's more like, do we get good outcomes at all?

Sure.

So let me say, first of all, like by doom, I mean something like by 2100, the world is in a state where I can see that almost everything that I value has been destroyed.

Maybe we're not like literally dead, but we've been forced to be upload in some very unfavorable conditions where it's just like some crappy lossy copy that never gets run.

And I feel like whatever dynamics are in charge of our civilization are just not going to optimize for anything that seems like it's going to be valuable.

And I guess I would say something like 70 to 80%, I think, just because

Again, we're up against competition.

I think by my standards, solving or avoiding this kind of fate looks like radically different outcomes than any other being or group of beings has had in history.

From my point of view, every animal has been in a situation where it has to either evolve into something unrecognizable and morally alien to it or die.

And we're by default in that situation too.

And by default, we end up being replaced by something that's more competitive than us and is like,

probably very morally alien and to us just seems like, again, cares about growth and nothing important.

It could be the case.

There's a small chance that just if we allow competition to flourish, that there's a bunch of amazing beings having awesome lives.

And I'm like, actually, that's really cool, even though I don't get to be part of it.

But I guess I'm very parochial in the sense that I'm like,

Me and my family, if we all die, that's just so bad that I almost consider that doom.

If most of humanity is in a similar situation.

So if it is just that we have runaway competition and we get replaced by some