Stephen McAleese

Quote The gap between before and after is the same curse that makes so many space probes fail.

After we launch them, probes go high and out of reach, and a failure, despite all careful theories and tests, is often irreversible.

Launching a space probe successfully is difficult because the real environment of space is always somewhat different to the test environment and issues are often impossible to fix after launch.

For ASI alignment, the gap before is our current state where the AI is not yet dangerous but our alignment theories cannot be truly tested against a superhuman adversary.

After the gap, the AI is powerful enough that if our alignment solution fails on the first try, we will not get a second chance to fix it.

Therefore, there would only be one attempt to get ASI alignment right.

Subheading Nuclear Reactors The authors describe the Chernobyl nuclear accident in detail and describe 4.

Engineering curses that make building a safe nuclear reactor and solving the ASI alignment problem difficult.

Speed.

Nuclear reactions and AI actions can occur much faster than human speed making it impossible for human operators to react and fix these kinds of issues when they arise.

Narrow margin for error.

In a nuclear reactor the neutron multiplication factor needs to be around 100% and it would fizzle out or explode if it were slightly lower or higher.

In the field of AI, there could be a narrow margin between a safe AI worker and one that would trigger an intelligence explosion.

Self-amplification.

Nuclear reactors and AIs can have self-amplifying and explosive characteristics.

A major risk of creating an ASI is its ability to recursively self-improve.

The curse of complications.

Both nuclear reactors and AIs are highly complex systems that can behave in unexpected ways.

Subheading.

Computer security.