Today’s EpisodeA developer posted this workflow in March, and it is the clearest picture of where PM is heading that I’ve seen all year.Rasty Turek spent the past year building with coding agents, and he mapped how his process changed over that time. He reckons he now spends around 90% of his time on evals. His eval started as QA, and then it became the spec.Great, now everyone agrees evals are important and will become indispensable for PMs going forward. But there is very little on how to write one.That changes today.I’ve now done 6 episodes on evals, and all of them start with an agent that is running and failing. So what do you do on day 0?Daniel McKinnon was a PM on the Llama models at Meta, a boomerang who spent around 7 years there in total. He sat on Facebook’s central AI team for the entirety of its existence. He wrote enterprise evals for Gemini, Llama, and Ray-Ban Meta.His first job at Meta was on the speech recognition team. He had to figure out how to check whether the models were any good. They weren’t called evals back then. But he’s been writing them for his entire career anyway.In this episode you’ll learn:* How to build an eval set from nothing* The floor-and-ceiling method for calibrating* How to score it and make the shipping callCheck it out:Please fill out this short survey on PM salaries.🆓 I’m doing a free webinar Thursday on getting AI PM interviews. Join me:The next cohort of my LandPMJob program starts in August. If you want my 1:1 coaching, sign up.----Check out the conversation on Apple, Spotify, and YouTube.Brought to you by:* SerpApi - Get started with SerpApi using 250 free credits.* Product Faculty - Get $550 off their #1 AI PM Certification with code AAKASH550C7* Ariso - Ship AI agents and features faster, with fewer regressions* Land PM Job - 12-week experience to master getting a PM job* Pendo - The #1 software experience management platform----Key Takeaways:1. An eval is a trivia question for the model - At its core, an eval is a prompt with a correct or plausibly correct answer plus a way to score whether the output is good. It is the clearest way to communicate what your product should do in the AI era.2. Offline evals catch problems before you ship - Test the model offline against a fixed prompt set before pushing to production. If it fails, you change the model, the prompt, or the approach before real users ever see it.3. The best eval sits between too easy and too hard - An eval that scores 100% gives your engineering team nothing to optimize. An eval that scores 0% is equally useless. Aim for a 25% to 50% success rate so there is room to run.4. Old benchmarks are already saturated - MMLU, HellaSwag, ARC and the rest were built for a simpler question-and-answer world. Frontier models now score effectively 100% on them, which is why you have to keep building new evals and throwing away old ones.5. Writing an eval is mechanical once you understand the problem - Come up with roughly 100 prompts that match the real distribution of tasks. The hard part is not the writing. It is deeply understanding the domain first.6. Subject matter expertise drives everything - The cystic fibrosis and congenital heart disease evals worked because Daniel understood the genetics, not because of any template or tool. There is no eval template the way there is a PRD template.7. Modern evals are agentic, not just Q&A - The genetics eval hands the agent a file with billions of variants and asks it to find the cause of a disease. This is a task, not a lookup, and it mirrors how real AI products now work.8. Find the model ceiling on purpose - The easy cystic fibrosis case gets solved by most models. The harder digenic congenital heart disease case exposes where even strong models fail. Knowing the ceiling is the point of the exercise.9. Sample multiple times before you trust a result - Models are non-deterministic. Run the same task several times so you understand the real distribution of outcomes rather than a single lucky or unlucky pass.10. Meta and Google build products very differently - Google is seen as more engineering-led, Meta as more product-led and far more aggressive culturally. Daniel worked on both Gemini and Llama and saw everything from Llama 3 highs to Llama 4 lows.----Where to find Daniel McKinnon* LinkedIn* X* Gamow LabsRelated contentPodcasts:* AI Evals with Hamel Husain and Shreya Shankar* How to Run Evals in Claude Code with Aparna Dhinakaran* Evals are the New PRD with Ankur GoyalNewsletters:* AI Evals for PMs: Everything You Need to Know to Get Started in 2026* AI PM’s Guide to LLM Judges* AI Evals Explained Simply----PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.