Dario Amodei

But they are a dangerous weapon to wield.

We should worry about them in the hands of autocracies, but also worry that because they are so powerful, with so little accountability, there is a greatly increased risk of democratic governments turning them against their own people to seize power.

AI surveillance

Sufficiently powerful AI could likely be used to compromise any computer system in the world and could also use the access obtained in this way to read and make sense of all the world's electronic communications or even all the world's in-person communications if recording devices can be built or commandeered.

It might be frighteningly plausible to simply generate a complete list of anyone who disagrees with the government on any number of issues, even if such disagreement isn't explicit in anything they say or do.

A powerful AI looking across billions of conversations from millions of people could gauge public sentiment, detect pockets of disloyalty forming, and stamp them out before they grow.

This could lead to the imposition of a true panopticon on a scale that we don't see today, even with the CCP.

AI propaganda.

Today's phenomena of AI psychosis and AI girlfriends suggest that even at their current level of intelligence, AI models can have a powerful psychological influence on people.

Much more powerful versions of these models, that were much more embedded in and aware of people's daily lives and could model and influence them over months or years, would likely be capable of essentially brainwashing many, most, people into any desired ideology or attitude, and could be employed by an unscrupulous leader to ensure loyalty and suppress dissent, even in the face of a level of repression that most populations would rebel against.

Today people worry a lot about, for example, the potential influence of TikTok as CCP propaganda directed at children.

I worry about that too, but a personalized AI agent that gets to know you over years and uses its knowledge of you to shape all of your opinions would be dramatically more powerful than this.

Strategic decision-making.

A country of geniuses in a data center could be used to advise a country, group, or individual on geopolitical strategy, what we might call at a virtual Bismarck.

It could optimize the three strategies above for seizing power, plus probably develop many others that I haven't thought of, but that a country of geniuses could.

Diplomacy, military strategy, R&D, economic strategy, and many other areas are all likely to be substantially increased in effectiveness by powerful AI.

Many of these skills would be legitimately helpful for democracies.

We want democracies to have access to the best strategies for defending themselves against autocracies, but the potential for misuse in anyone's hands still remains.

Having described what I am worried about, let's move on to who.

I am worried about entities who have the most access to AI, who are starting from a position of the most political power, or who have an existing history of repression.