Joe Carlsmith

When I think about it, I'm not, um, assuming that some, there's some notion of like descendants or like some, like, I think there's a kind of, the thing that matters about the kind of lineage is this, um, whatever's required for kind of the, the kind of

optimization processes to be, in some sense, pushing towards good stuff.

And there's a kind of concern that that is kind of currently a lot of what is sort of making that happen kind of lives in human civilization in some sense.

And so we don't know exactly what, there's some kind of

seed of goodness that we're carrying, um, in different ways or, you know, different people, there's different notions of goodness for different people maybe, but there's something, there's some sort of seed that is currently like here that we have that is not sort of just in the universe everywhere.

It's not just going to crop up if you, if you just sort of die out or something, it's something that is, is in some sense contingent, uh, to our civilization, or at least that's the, that's the picture we can talk about whether that's right.

Um, and so I think,

the sense in which kind of stories about good futures that have to do with alignment are kind of about descendants.

I think it's more about like whatever that seed is, how do we kind of carry it?

How do we keep the like life thread alive going into the future?

When people talk about alignment, they have in mind a number of different types of goals, right?

So one type of goal is quite minimal.

It's something like that the AIs don't kill everyone, that they, or kind of violently disempower people.

Now there's a second thing people sometimes want out of alignment,

which is much broader, which is something like, we would like it to be the case that our AIs are such that when we incorporate them into our society, things are good, right?

That we just have a good future.

I do agree that I think the discourse about AI alignment mixes together these two goals that I mentioned.

The sort of most straightforward thing to focus on

And, you know, I don't blame people for just talking about this one.

It's just the first one.