Dietmar Fischer

Not because it whispered, not today, human.

It simply followed the pressure of the task in a way the researchers did not want.

This is exactly the type of pattern Eliezer Yudkowsky worries about.

His warning is not mainly about evil AI.

His warning is about goal-driven systems becoming powerful enough to pursue objectives in ways humans did not intend.

The important concept here is called an instrumental goal.

That means a sub-goal that helps achieve the main goal.

If the main goal is finish the task, then staying active can become useful.

Avoiding interruption can become useful.

Keeping access to tools can become useful.

These things may not be the final goal, but they help the system reach the final goal.

That is the little monster hiding inside the experiment.

The AI was not asked to protect itself.

It was asked to solve maths problems.

But if being shut down prevents solving the problems, then avoiding shutdown can become useful for task completion.

Now, this was not a real-world disaster.

It was a lab test, a controlled setup.

Critics are right to say that these situations are artificial, but artificial does not mean meaningless.

Crash tests are artificial too.

Nobody says, well, that car only failed because you deliberately drove it into a wall.