Stephen McAleese

For example, Hitler was a real person, and he was wildly anti-Semitic.

Every single item on their list that supposedly provides evidence of alien drives is more consistent with a human drives theory.

In other words, their evidence effectively shows the opposite conclusion from the one they claim it supports.

Finally, the post does not claim that AI is risk-free.

Instead it argues for an empirical approach that studies and mitigates problems observed in real-world AI systems.

The most plausible future risks from AI are those that have direct precedence in existing AI systems, such as sycophantic behavior and reward hacking.

These behaviors are certainly concerning, but there's a huge difference between acknowledging that AI systems pose specific risks in certain contexts and concluding that AI will inevitably kill all humans with very high probability.

End quote.

Subheading.

Arguments against the evolution analogy.

Several critics of the book and its arguments criticize the book's use of the human evolution analogy as an analogy for how ASI would be misaligned with humanity and argue that it is a poor analogy.

Instead they argue that human learning is a better analogy.

The reason why is that both human learning and AI training involve directly modifying the parameters responsible for human or AI behavior.

In contrast, human evolution is indirect.

Evolution only operates on the human genome that specifies a brain's architecture and reward circuitry.

Then all learning occurs during a person's lifetime in a separate inner optimization process that evolution cannot directly access.

In the essay Unfalsifiable Stories of Doom, the authors argue that because gradient descent and the human brain both operate directly on neural connections, the resulting behavior is far more predictable than the results of evolution.

Quote

A critical difference between natural selection and gradient descent is that natural selection is limited to operating on the genome, whereas gradient descent has granular control over all parameters in a neural network.

The genome contains very little information compared to what is stored in the brain.