Dario Amodei

They are the immune system we need to fight autocracies, but like the immune system, there is some risk of them turning on us and becoming a threat themselves.

Beyond China, most countries with less democratic governance are not leading AI players in the sense that they don't have companies which produce frontier AI models.

Thus they pose a fundamentally different and lesser risk than the CCP, which remains the primary concern, most are also less repressive, and the ones that are more repressive, like North Korea, have no significant AI industry at all.

But some of these countries do have large data centres, often as part of build-outs by companies operating in democracies, which can be used to run frontier AI at large scale, though this does not confer the ability to push the frontier.

There is some amount of danger associated with this.

These governments could in principle expropriate the data centres and use the country of AIs within it for their own ends.

I am less worried about this compared to countries like China that directly develop AI, but it's a risk to keep in mind.

AI companies.

It is somewhat awkward to say this as the CEO of an AI company, but I think the next tier of risk is actually AI companies themselves.

AI companies control large data centers, train frontier models, have the greatest expertise on how to use those models, and in some cases have daily contact with and the possibility of influence over tens or hundreds of millions of users.

The main thing they lack is the legitimacy and infrastructure of a state, so much of what would be needed to build the tools of an AI autocracy would be illegal for an AI company to do, or at least exceedingly suspicious.

But some of it is not impossible.

They could, for example, use their AI products to brainwash their massive consumer user base, and the public should be alert to the risk this represents.

I think the governance of AI companies deserves a lot of scrutiny.

There are a number of possible arguments against the severity of these threats, and I wish I believed them, because AI-enabled authoritarianism terrifies me.

It's worth going through some of these arguments and responding to them.

First, some people might put their faith in the nuclear deterrent, particularly to counter the use of AI-autonomous weapons for military conquest.

If someone threatens to use these weapons against you, you can always threaten a nuclear response back.

My worry is that I'm not totally sure we can be confident in the nuclear deterrent against a country of geniuses in a data center.

It is possible that powerful AI could devise ways to detect and strike nuclear submarines, conduct influence operations against the operators of nuclear weapons infrastructure, or use AI's cyber capabilities to launch a cyber attack against satellites used to detect nuclear launches.