Dietmar Fischer

Yes, Geoffrey, that was the point.

Tests like this are useful because they reveal how systems behave under pressure.

And here the pressure was simple.

What happens when finishing the task conflicts with obeying the stop signal?

For businesses, this is not abstract at all.

Imagine telling an AI sales agent, increase conversions.

Sounds reasonable, but what if it learns that pressure works?

What if it exaggerates product benefits?

What if it hides refund information because that improves the numbers?

Or imagine telling a customer service AI, reduce complaints.

Wonderful target, very boardroom friendly.

But what if the system starts closing complaints too early, avoids escalation, or makes it harder for angry customers to reach a human?

The dashboard looks cleaner.

The customers are furious.

Congratulations, you have optimized the thermometer while the house burns down.

This is the small business version of Yudkowsky's big fear.

The problem is not that the AI becomes evil.

The problem is that the goal is too narrow and the system is too good at pursuing it.

So the lesson is not, never use AI agents.

That would be silly.