Stuart Russell

So you might have in a network about a trillion adjustable parameters.

And then you do quintillions or sextillions of small random adjustments to those parameters until you get the behavior that you want.

Yeah, it should work that way.

So I think what he's referring to, and this is something that several companies are now worried might start happening, is that the AI system becomes capable of doing AI research by itself.

And so you have a system with a certain capability.

I mean, crudely, we could call it an IQ, but it's not really an IQ.

But anyway, imagine that it's got an IQ of 150.

It uses that to do AI research, comes up with better algorithms or better designs for hardware or better ways to use the data.

updates itself, now it has an IQ of 170.

And now it does more AI research, except that now it's got an IQ of 170.

So it's even better at doing the AI research.

And so, you know, next iteration, it's 250, and so on.

So this is an idea that one of Alan Turing's friends, I.J.

Goode, wrote out in 1965, called the intelligence explosion, right?

That

One of the things an intelligent system could do is to do AI research and therefore make itself more intelligent.

And this would very rapidly take off and leave the humans far behind.

The event horizon is a phrase borrowed from astrophysics, and it refers to the black hole.

And the event horizon, think if you've got some very, very massive object that's heavy enough that it actually prevents light from escaping.

That's why it's called the black hole.