Eliezer Yudkowsky

And is it real?

Probably not.

But nobody knows what's going on in there.

It's part of a process where these things are not happening in a way where we...

somebody figured out how to make an AI care, and we know that it cares, and we can acknowledge its caring now.

It's being trained by this imitation process, followed by reinforcement learning on human feedback.

And we're trying to point it in this direction, and it's pointed partially in this direction, and nobody has any idea what's going on inside it.

And if there was a tiny fragment of real caring in there, we would not know.

It's not even clear what it means exactly.

And

Things are a clearer cut in science fiction.

Except that's probably not yet.

It probably isn't happening right now.

We are boiling the frog.

We are seeing increasing signs, bit by bit.

But not like spontaneous signs, because people are trying to train the systems to do that using imitative learning.

And the imitative learning is like spilling over and having side effects.

And the most photogenic examples are being posted to Twitter.

rather than being examined in any systematic way.

So when you are boiling a frog like that, you're going to get, like, first is going to come the Blake Lemoines.