Stephen McAleese

Finally the authors compare ASI alignment to computer security.

Both fields are difficult because designers need to guard against intelligent adversaries that are actively searching for flaws in addition to standard system errors.

Heading.

Counter-arguments to the book.

In this section I describe some of the best critiques of the book's claims and then distill them into three primary counter-arguments.

Subheading.

Arguments that the book's arguments are unfalsifiable.

Some critiques of the book such as the essay Unfalsifiable Stories of Doom argue that the book's arguments are unfalsifiable, not backed by evidence, and are therefore unconvincing.

Obviously since ASI doesn't exist, it's not possible to provide direct evidence of misaligned ASI in the real world.

However, the essay argues that the book's arguments should at least be substantially supported by experimental evidence and make testable and falsifiable predictions about AI systems in the near future.

Additionally, the post criticizes the book's extensive usage of stories and analogies rather than hard evidence and even compares its arguments to theology rather than science.

Quote

What we mean is that Y and SS methods resemble theology in both structure and approach.

Their work is fundamentally untestable.

They develop extensive theories about non-existent, idealized, ultra-powerful beings.

They support these theories with long chains of abstract reasoning rather than empirical observation.

They rarely define their concepts precisely, opting to explain them through allegorical stories and metaphors whose meaning is ambiguous.

Although the book does mention some forms of evidence, the essay argues that the evidence actually refutes the book's core arguments and that this evidence is used to support pre-existing pessimistic conclusions.

But in fact, none of these lines of evidence support their theory.

All of these behaviors are distinctly human, not alien.