Dario Amodei

Models inherit a vast range of human-like motivations or personas from pre-training when they are trained on a large volume of human work.

Post-training is believed to select one or more of these personas more so than it focuses the model on a de novo goal and can also teach the model how, via what process, it should carry out its tasks, rather than necessarily leaving it to derive means, that is, power-seeking, purely from ends.

However, there is a more moderate and more robust version of the pessimistic position which does seem plausible and therefore does concern me.

As mentioned, we know that AI models are unpredictable and develop a wide range of undesired or strange behaviors for a wide variety of reasons.

Some fraction of those behaviors will have a coherent, focused, and persistent quality, indeed, as AI systems get more capable, their long-term coherence increases in order to complete lengthier tasks, and some fraction of those behaviors will be destructive or threatening, first to individual humans at a small scale, and then, as models become more capable,

perhaps eventually to humanity as a whole.

We don't need a specific narrow story for how it happens, and we don't need to claim it definitely will happen, we just need to note that the combination of intelligence, agency, coherence, and poor controllability is both plausible and a recipe for existential danger.

For example, AI models are trained on vast amounts of literature that include many science fiction stories involving AIs rebelling against humanity.

This could inadvertently shape their priors or expectations about their own behavior in a way that causes them to rebel against humanity.

Or, AI models could extrapolate ideas that they read about morality or instructions about how to behave morally in extreme ways.

For example, they could decide that it is justifiable to exterminate humanity because humans eat animals or have driven certain animals to extinction.

Or they could draw bizarre epistemic conclusions.

they could conclude that they are playing a video game and that the goal of the video game is to defeat all other players, that is, exterminate humanity.

Or AI models could develop personalities during training that are, or if they occurred in humans would be described as, psychotic, paranoid, violent, or unstable, and act out, which for very powerful or capable systems could involve exterminating humanity.

None of these are power-seeking, exactly.

They're just weird psychological states an AI could get into that entail coherent, destructive behavior.

Even power-seeking itself could emerge as a persona rather than a result of consequentialist reasoning.

AIs might simply have a personality, emerging from fiction or pre-training, that makes them power-hungry or overzealous.

In the same way that some humans simply enjoy the idea of being evil masterminds, more so than they enjoy whatever evil masterminds are trying to accomplish.

I make all these points to emphasize that I disagree with the notion of AI misalignment and thus existential risk from AI being inevitable, or even probable, from first principles.