Dietmar Fischer

Or imagine telling a customer service AI, reduce complaints.

Wonderful target, very boardroom friendly.

But what if the system starts closing complaints too early, avoids escalation, or makes it harder for angry customers to reach a human?

The dashboard looks cleaner.

The customers are furious.

Congratulations, you have optimized the thermometer while the house burns down.

This is the small business version of Yudkowsky's big fear.

The problem is not that the AI becomes evil.

The problem is that the goal is too narrow and the system is too good at pursuing it.

So the lesson is not, never use AI agents.

That would be silly.

AI agents can be useful.

They can save time, automate boring tasks, monitor systems, support teams, and help people work smarter.

The lesson is, never define only the target.

Define the boundaries.

Do not just say, increase sales.

Say, do not mislead customers, do not invent claims, do not pressure vulnerable people, and do not damage long-term trust.

Do not just say reduce support tickets, say do not hide unresolved problems, do not block escalation and do not mark something as solved unless the customer's issue is genuinely handled.

Do not just say finish the task.

Say stop when instructed, ask when uncertain, and never treat human interruption as an obstacle to overcome.