Stephen Wolfram

There's a kind of meaningful calculus of I want X, and that has various consequences.

I mean, I'm not sure.

I haven't thought this through properly, but I think a concept like is so-and-so a good person, is that true or not?

That's a mess.

I mean, that particular thing, it's kind of like, you know, we're going into kind of the ethics of what counts as good, so to speak.

And, you know, what do we think is right and so on.

And I think that's a thing which, you know, one feature is we don't all agree about that.

There's no theorems about kind of, you know, there's no theoretical framework that says this is...

This is the way that ethics has to be.

Yeah.

Well, I think one of the things with AIs is it's one thing to wipe out that AI that has no owner.

You can easily imagine an AI kind of hanging out on the internet without having any particular owner or anything like that.

And then you say, well, what harm does it... It's okay to get rid of that AI.

Mm-hmm.

Of course, if the AI has 10,000 friends who are humans and all those 10,000 humans will be incredibly upset that this AI just got exterminated, it becomes a slightly different, more entangled story.

But yeah, no, I think that this question about what do humans agree about

It's, you know, there are certain things that, you know, human laws have tended to consistently agree about.

You know, there have been times in history when people have sort of gone away from certain kinds of laws, even ones that we would now say, how could you possibly have not done it that way?

You know, that just doesn't seem right at all.

But I think, I mean, this question of what