Nathaniel Whittemore

As I mentioned before, a lot of the chatter is around the sandbox breakout and the emailing of the researcher while they were eating in the park.

And one less theoretical safety concern that came up was a note fairly deep in the security card where Anthropic admitted to having accidentally trained against the chain of thought for Opus 4-6, Sonnet 4-6, and Mythos for 8% of reinforcement learning.

So let me be clear, at a minimum, any and all reassuring evidence from their chain of thoughts is worthless.

They are hopelessly corrupted.

When Dan McAdory writes, Zvi, can you explain why this is bad?

Matt Hurd summed up, If the chain of thought is trained against, selective pressure will teach it to hide unwanted behavior from appearing in its chain of thought, making its chain of thought less faithful to its internal process, making chain of thought observation less effective at detecting unwanted behavior.

Zvi actually wrote about this extensively back in March of 2025 in a blog post called The Most Forbidden Technique.

He writes, The most forbidden technique is training an AI using interpretability techniques.

An AI produces a final output X via some method M. You can analyze M using technique T to learn what the AI is up to.

You could train on that.

Never do that.

You train on X.

i.e.

the final output.

Only X. Never M, never T. Why?

Because T is how you figure out when the model is misbehaving.

If you train on T, you are training the AI to obfuscate its thinking and defeat T. You will rapidly lose your ability to know what is going on in exactly the ways you most need to know what's going on.

Another thing that Anthropic team members discussed was the exhibited internal behavior of Cloud Mythos.

For example, Jack Lindsay writes, early versions of Mythos preview often exhibited over-eager and or destructive actions.

The model bulldozing through obstacles to complete a task in a way the user wouldn't want.