Joe Carlsmith

thought about AIs, how we treated our AIs.

And we end up looking back with a kind of moral horror at what we were doing.

So, you know, we end up thinking, you know, we were thinking about these things centrally as like products, as tools.

But in fact, we should have been foregrounding much more the sense in which they might be moral patients or were moral patients at some level of sophistication, that we were kind of treating them in the wrong way.

We were just

acting like we could do whatever we want.

We could delete them, subject them to arbitrary experiments, kind of alter their minds in arbitrary ways.

And then we end up looking back in the light of history at that as a kind of serious and kind of grave moral error.

Those are scenarios I think about a lot in which we have regrets.

I don't think they quite fit the bill of what you just said.

I think it sounds to me like the thing you're thinking is something more like we end up feeling like, gosh,

we wish we had paid no attention to the motives of our AIs, that we'd thought not at all about their impact on our society as we incorporated them.

And instead, we had pursued a, let's call it a kind of maximize for brute power option, which is just kind of make a beeline for whatever is just the most powerful AI you can, and don't think about anything else.

Cool.

So I think there's a bunch of different things to potentially unpack there.

One kind of conceptual point that I want to name off the bat, I don't think you're necessarily kind of making a mistake in this vein, but I just want to name it as like a possible mistake in this vicinity is I think we don't want to engage in the following form of reasoning.

Let's say you have two entities.

One is in the role of creator and one is in the role of creation.

And then we're positing that there's this kind of misalignment relation between them, whatever that means, right?

And