Martin Wolf

The devil promises infinite wealth and power to Faust in return for his immortal soul, which is in some sense his meaning.

And he makes the bargain and it doesn't end well.

So we have been fantastically innovative and creative, and we have in the process made ourselves vastly wealthier and wiser in some obvious ways.

But we're always skirting the brink of catastrophe as a result.

And the last...

thing that really motivated people around those lines was nuclear weapons.

People like Einstein got terribly frightened about what they created.

Well, I think of AI that way.

We have made a Faustian bargain and we have created a servant, quote unquote, with the capacity to replace us or even in some way to dominate us.

Where this will end up, I don't know.

But it seems to me there are a number of absolutely terrifying dangers.

I think, and I thought for the first time, I read about it three or four years ago, that it was going to prove completely unregulatable.

It would, the competition process and so forth, it would make institutions that used it unaccountable in some profound way because the decisions were not being taken by anyone you can hold to account.

It would create transformation in our society, in our economy, and in our sense of ourselves, which was unavoidably just

of an order of magnitude different from anything that had come before.

And...

In the short to medium run, a lot of this would seem very helpful, very encouraging.

We will enjoy it.

We will be able to do things we couldn't do otherwise.

But if we do get to artificial general intelligence, and it's absolutely clear that the machine dominates humanity in terms of what it can do intellectually in every possible way,