Ilya Sutskever

So this is an area where different people have different intuitions.

I think it will be very powerful for sure.

I think that what I think is most likely to happen is that there will be multiple...

such AIs being created roughly at the same time.

I think that if the cluster is big enough, if the cluster is literally continent-sized, that thing could be really powerful indeed.

If you literally have a continent-sized cluster, those AIs can be very powerful.

And all I can tell you

is that if you're talking about extremely powerful AIs, like truly dramatically powerful, then yeah, it would be nice if they could be restrained in some ways, or if there was some kind of an agreement or something.

Because I think that if you are saying, hey, what is the concern of superintelligence?

What is one way to explain the concern?

If you imagine a system that is sufficiently powerful, like really sufficiently powerful, and you could say, okay, you need to do something sensible, like care for sentient life, let's say, in a very single-minded way.

might not like the results that's really what it is and so maybe by the way the answer is that you do not build a single you do not build an rl agent in the usual sense and actually i'll point i'll point several things out i think human beings are a semi-rel agent you know we pursue a reward and then the emotions or whatever make us tire out of the reward we pursue a different reward

The market is like, kind, it's like a very short sighted kind of agent evolution is the same.

Evolution is very intelligent in some ways, but very dumb in other ways.

The government has been designed to be a never ending fight between three parts, which has an effect.

So I think things like this.

Another thing that makes this discussion difficult is that we are talking about systems that don't exist, that we don't know how to build.

That's the other thing.

And that's actually my belief.

I think what people are doing right now will go some distance and then peter out.