Joe Carlsmith

And I don't think we need to be like, my mom has to have a utility function with the dog and she has to have a consistent valuation of all the houses or whatever.

But it's still the case that her planning and her agency exerted in the world resulted in her having this house, having this dog.

And I think it is plausible that as our kind of scientific and technological power advances,

more and more stuff will be kind of explicable in that way.

Right.

That, you know, if you look and you're like, why is this man on the moon?

Right.

How did that happen?

And it's like, well, like there was a whole cognitive process.

There's a whole like planning, uh, apparatus.

Now in this case, it wasn't like localized in a single mind, but like there was a whole thing such that man on the moon.

Right.

Um, and I think like, we'll see a bunch more of that.

And the AI is, uh,

will be, I think, doing a bunch of it.

And so that's the thing that seems more real to me than kind of utility functions.

Yeah, I agree.

I think there's a few things going on there.

So one is that I do think that, um, even if you're engaged in this ontology of kind of carving up the world into different like agencies, um,

at the least you don't want to kind of assume that they're all like unitary or like not overlapping or like, right.