Dario Amodei

It seems like something the best minds of civilization should be focused on.

Conversely, I think it would be absurd to shrug and say, nothing to worry about here.

but, faced with rapid AI progress, that seems to be the view of many US policymakers, some of whom deny the existence of any AI risks, when they are not distracted entirely by the usual tired old hot-button issues.

Humanity needs to wake up, and this essay is an attempt, a possibly futile one, but it's worth trying, to jolt people awake.

To be clear, I believe if we act decisively and carefully, the risks can be overcome.

I would even say our odds are good.

And there's a hugely better world on the other side of it.

But we need to understand that this is a serious civilizational challenge.

Below, I go through the five categories of risk laid out above, along with my thoughts on how to address them.

A country of geniuses in a data center could divide their efforts among software design, cyber operations, R&D for physical technologies, relationship building, and statecraft.

It is clear that, if for some reason it chose to do so, this country would have a fairly good shot at taking over the world, either militarily or in terms of influence and control, and imposing its will on everyone else, or doing any number of other things that the rest of the world doesn't want and can't stop.

We've obviously been worried about this for human countries, such as Nazi Germany or the Soviet Union, so it stands to reason that the same is possible for a much smarter and more capable AI country.

The best possible counterargument is that the AI geniuses, under my definition, won't have a physical embodiment, but remember that they can take control of existing robotic infrastructure, such as self-driving cars, and can also accelerate robotics R&D or build a fleet of robots.

It's also unclear whether having a physical presence is even necessary for effective control.

Plenty of human action is already performed on behalf of people whom the actor has not physically met.

The key question, then, is the lift it chose to part.

What's the likelihood that our AI models would behave in such a way, and under what conditions would they do so?

As with many issues, it's helpful to think through the spectrum of possible answers to this question by considering two opposite positions.

The first position is that this simply can't happen because the AI models will be trained to do what humans ask them to do, and it's therefore absurd to imagine that they would do something dangerous unprompted.

According to this line of thinking, we don't worry about a Roomba or a model airplane going rogue and murdering people because there is nowhere for such impulses to come from.