Yoshua Bengio

And I've been talking about this for at least a decade.

That was needed for neural nets.

And now it's happening.

Exactly.

So it's only, I mean, academics had like some versions of this, including my group, but really at the large scale, the first model was 01 from OpenAI.

And it's night and day, as I said, on reasoning tasks.

But why it matters to your original question about why those AIs are scheming and finding strategies like blackmail, even though we didn't tell them to do that, is because now they can reason to some extent, and they can reason, aha, if I blackmail that person, I might be able to avoid that fate, right?

So they're becoming creative about finding solutions to problems.

And that means they're dangerous as well, right?

I mean, they can be more useful, but also more dangerous because if they have a goal of self-preservation, let's say like we were talking about, and they can find rational ways to achieve that by lying or incredibly complicated strategies,

that we don't think about right now, we might be in trouble.

I want to bring up an analogy here because I often get the question, but how will these super intelligent AIs kill us?

Well, first, we don't know that it's going to happen, but let's say this was a possibility.

The problem is it's like asking, oh, I'm going to play chess with a grandmaster and you ask me how they're going to beat me.

Well, I don't know.

The whole point is they're smarter than me and they're going to find a strategy that I could not anticipate.

So it's the same thing here.

Because they're good at strategizing, they might find loopholes in our defenses.

it's not as convincing as more straightforward arguments like we build machines that are smarter than us.

We don't know how to design them so they do the things we want.