Sean Carroll

I don't even think that there is a coherent idea called ASI, as I was saying before.

There's many different levels of intelligence, directions of intelligence, and we're not helping ourselves where we blur all those important distinctions and lump them together in one thing.

We should take seriously

what we actually have and might create, not just project some vague ideas about superness onto the concept of intelligence.

But okay, having said all that, I think that questions like this are in the genre of questions that become very difficult because you're saying,

Okay, I have this thing that I think is very, very unlikely, but if it happens, it would have a huge, huge, huge impact, right?

And I mean very unlikely and very huge impact.

So I need to multiply the percent chance that it happens by the impact, and I have to sort of weigh that against things that are quite likely and have much smaller impact.

I think that in general, questions like those are really, really hard.

It's not hard in principle to do the math.

What's hard to do is to know whether or not doing the math is the right thing to do.

And it's really hard to understand the error bars on your math.

Is something that is unlikely a 1% chance or a 10 to the minus 100 chance?

Those are very, very different numbers.

Is the impact really extinguishing life on Earth or is it costing us a few billion dollars in revenue?

Very, very different outcomes.

So I think that my general strategy for these questions is to be somewhat incrementalist about them.

That is to say, not to really base my actions on these best case or worst case scenarios in the relatively far future, but to be prudent about the direction we're moving in, step to step.

I think that we have much more

control and understanding of what short-term possibilities are than we do long-term possibilities.