Show notes
Today’s episodeMany of the smartest AI teams I know are running their evals on Arize. Teams at Uber, Booking.com, Pepsi, and others.It’s become one of the most important skills for PMs. I already had on the CEO of Braintrust, Hamel Husain and Shreya Shankar, and Ankit Shukla.Today I’m adding to this knowledge base on evals with a masterclass on evals in Claude Code.Aparna Dhinakaran is the founder of Arize. She’s also their CPO. And she gives a masterclass in how to run all of your evals through Claude Code.So if you want to do AI evals like the best, like Uber, like Booking.com, check out this episode. For anyone in building in Claude Code, it’s a doozy.If a candidate did this in an interview, Aparna said she would hire them on the spot.----Check out the conversation on Apple, Spotify, and YouTube.Brought to you by:* Superhuman - The fastest email experience ever made* Sign up and get 1-month free of Superhuman Mail with my link: superhuman.com/akash (given by brand - Kartik)* Land PM Job - My 12-week AI PM + Job Search Course, first 10 enrollees get a FREE 30-min 1:1 consultation* Vanta - Automate your compliance. Close deals faster* Product Faculty - Get $550 off their #1 AI PM Certification with code AAKASH550C7* Bolt - Ship AI-powered products 10x faster----If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.Do you want to become an AI PM? I’ve created a course for you. Starts soon.----Key Takeaways:1. Trace before you eval - A trace is the full step-by-step playback of what your agent did. Without it, you have no evidence base for evals. Every LLM call, every tool call, every intermediate output needs to be visible before you write a single eval.2. A span is your unit of evaluation - A span is one discrete step inside a trace. Evals run at the span level, not the trace level. "Did this specific scoring step get the priority right?" is a more useful question than "was the whole run good?"3. Instrumentation is now a one-command job - Claude Code's instrumentation skills can set up observability for your agent automatically. Arize Phoenix's skill looks at your codebase, identifies the LLM calls and tool calls, and wires them to the tracing layer. No engineering support required.4. The vibe eval is a draft, not a verdict - An LLM can suggest what your evals should test by looking at your traces. That suggestion will not know your bug-first policy, your comp logic, or your definition of "critical." Treat it as v0 and refine against your actual judgment.5. When evals fire, two things could be wrong - The agent produced a bad output. Or the eval is miscalibrated. Reading the flagged span yourself is the only way to know which one needs fixing. Both are normal. Both are good news.6. Evals drift and need regular realignment - Your priorities change. Your bug policy changes. Your product changes. An eval calibrated to last quarter will start misfiring this quarter. Regular alignment to human feedback is maintenance, not a failure.7. The self-improvement loop is already running at the best teams - Fetch all spans where evals fired. Group by failure category. Propose a specific prompt fix. Review and approve. Ship the new version. This loop runs on a schedule and requires a human at the approval step.8. Enterprise PMs: start with one internal agent - Not a customer-facing product. An internal tool that takes four hours off your week. Once you have it, you will naturally want to trace it. That is when observability starts to matter to you personally.9. The context graph is the enterprise unlock - Agents are only as useful as the context they have. Enterprise data lives in silos. The teams breaking through are building unified context layers that give one agent access to CRM, Gong, analytics, GitHub, and Slack.10. Product taste is still the alpha - Code is cheap now. Shipping speed is table stakes. The PMs who pull ahead are the ones with the sharpest judgment about what to build, and the loops that make their agents better every day.----Related contentPodcasts:* AI Evals with Hamel Husain and Shreya Shankar* Evals are the new PRD with Ankur Goyal* AI PM Crash Course with Aman KhanNewsletters:* AI Evals for PMs: Everything You Need to Know to Get Started in 2026* Your Complete AI PM Course & Career Roadmaps* AI PM’s Guide to LLM Judges----PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe