Anil Seth

We humans have a terrible track record in our ethical treatments of non-human animals and of other humans, and we don't want to make the same mistake again.

And this is one reason why even trying to build conscious AI is a very bad idea.

But if conscious AI is just an illusion created by design, as I think it is, then by extending rights to these systems, we'd be sacrificing our ability to control, to regulate them, and perhaps even to turn them off, and for no good reason at all.

And this is one reason why even AI that merely seems to be conscious is very dangerous for our society too.

And unlike real artificial consciousness, conscious-seeming AI is either already here or coming very, very soon.

There are other reasons we should avoid creating AI that seems to be conscious.

It makes us more psychologically vulnerable.

We might be more likely to do what an AI says if we believe that it really feels for us, that it really understands us, even if what it's telling us to do is very bad for us.

And finally, the very idea of conscious AI undermines our human nature.

The mirror of AI goes both ways.

We see ourselves in our algorithms,

but we also see our algorithms in ourselves.

And when we do, when we think of the mind as a collection of computations floating free from their basis in biology, we reduce and we diminish what it is to be a living, breathing human being in a real world.

Frankenstein, which Mary Shelley wrote when she was just 19, is often taken as a cautionary tale, a warning against the hubris of bringing something to life, a Promethean sin, like stealing fire from the gods.

The idea of conscious AI is a new Promethean dream wrapped up in a silicon rapture.

And if conscious AI is possible, then so is the prospect of uploading our own conscious minds and floating off to eternity in a silicon cloud of living or at least existing forever in the pristine circuits of some future supercomputer.

Now, the seductive power of this vision, of being at such a pivotal point in the history of life on Earth, I suppose it's understandable.

And back in the real world, talk of conscious AI does other things too.

It conjures this sense of technological wonder and magic which might keep share prices aloft and regulators at bay.

But we should resist.