Rob Wiblin

It might have such an easy time even advancing to mirror biology, mirror bacteria, that there's nothing really that we can do to improve our resilience meaningfully.

We're just toast no matter what.

I guess you don't share that view.

Why is that?

Why don't I share this with you?

So I think part of what's going on with this mentality that there's no biological countermeasures that you can have that would really constrain the kind of misaligned AI that we're worried about is because people, for a long time, people have been worried about this massive intelligence explosion, the kind of fume scenario where you go from human level to like vastly superhuman, super intelligence.

Overnight.

Yeah, overnight.

I guess originally, literally overnight.

I guess now people, even the most extreme people probably talk about weeks.

If that's how things go, then it might be the case that any kind of measure that you put in place, an AI that is just many, many times smarter than the whole of humanity put together, would be able to find some way around it and would be able to kill you one way or another.

Because it's just making science... Maybe you don't agree with that.

We'll come back to that.

But it would be able to make so many scientific advances that you're just not going to be able to stop it.

We don't know whether that will happen at all.

We don't know whether there'll be an intelligence explosion at all.

We could be in a world where the feedback loop is too weak.

We basically just have a gradual increase in capabilities all the way through.

In that case, at any point in time,

ability to do damage for a rogue AI, for a misaligned AI, is only going to be somewhat above the level of knowledge that humans have.