Eliezer Yudkowsky

And it's probably not.

Nobody actually knows.

We don't have any other guardrails.

We don't have any other tests.

We don't have any lines to draw on the sand and say like, well, when we get this far, we will start to worry about what's inside there.

So if it were up to me, I would be like, OK, like this far, no further time for the summer of AI where we have planted our seeds and now we like wait and reap the rewards of the technology we've already developed and don't do any larger training runs than that, which, to be clear, I realize requires more than one company agreeing to not do that.

That would take decades.

Like, having any idea of what's going on in there, people have been trying for a while.

I mean, there's a whole bunch of different sub-questions here.

There's the question of...

Like, is there consciousness?

Is there qualia?

Is this an object of moral concern?

Is this a moral patient?

Like, should we be worried about how we're treating it?

And then there's questions like, how smart is it exactly?

Can it do X?

Can it do Y?

And we can check how it can do X and how it can do Y.

Unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the Internet, which means that when it talks about being self-aware, we don't know to what extent it is repeating back what it has previously been trained on for discussing self-awareness.