Anthropic deprecated Sonnet 3.5. Some of Xelix's pipelines migrated smoothly. Others broke — and customers noticed within hours. What separated the two? Evaluation. Paul Solomon and James Price Farr have spent 5+ years building AI systems that process millions of invoices for enterprise customers. In this episode, they share the evaluation-first framework that now saves them every time a model changes, an orchestration layer fails, or an agent picks the wrong tool. We cover: why monitoring tool calls matters more than monitoring outputs, the 3 tiers of automation (and why most teams jump to agents too early), how extended thinking on Bedrock actually reduced their costs, why steering files hit 100% accuracy where prompts alone capped at 82%, and how to earn customer trust by starting at 10% automation and scaling up only after you prove reliability. Recorded live at AWS Summit London.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related