How I AI

Evals, error analysis, and better prompts: A systematic approach to improving your AI products | Hamel Husain (ML engineer)

13 Oct 2025

Audio

Description

Hamel Husain, an AI consultant and educator, shares his systematic approach to improving AI product quality through error analysis, evaluation frameworks, and prompt engineering. In this episode, he demonstrates how product teams can move beyond “vibe checking” their AI systems to implement data-driven quality improvement processes that identify and fix the most common errors. Using real examples from client work with Nurture Boss (an AI assistant for property managers), Hamel walks through practical techniques that product managers can implement immediately to dramatically improve their AI products.What you’ll learn:1. A step-by-step error analysis framework that helps identify and categorize the most common AI failures in your product2. How to create custom annotation systems that make reviewing AI conversations faster and more insightful3. Why binary evaluations (pass/fail) are more useful than arbitrary quality scores for measuring AI performance4. Techniques for validating your LLM judges to ensure they align with human quality expectations5. A practical approach to prioritizing fixes based on frequency counting rather than intuition6. Why looking at real user conversations (not just ideal test cases) is critical for understanding AI product failures7. How to build a comprehensive quality system that spans from manual review to automated evaluation—Brought to you by:GoFundMe Giving Funds—One account. Zero hassle: https://gofundme.com/howiaiPersona—Trusted identity verification for any use case: https://withpersona.com/lp/howiai—Where to find Hamel Husain:Website: https://hamel.dev/Twitter: https://twitter.com/HamelHusainCourse: https://maven.com/parlance-labs/evalsGitHub: https://github.com/hamelsmu—Where to find Claire Vo:ChatPRD: https://www.chatprd.ai/Website: https://clairevo.com/LinkedIn: https://www.linkedin.com/in/clairevo/X: https://x.com/clairevo—In this episode, we cover:(00:00) Introduction to Hamel Husain(03:05) The fundamentals: why data analysis is critical for AI products(06:58) Understanding traces and examining real user interactions(13:35) Error analysis: a systematic approach to finding AI failures(17:40) Creating custom annotation systems for faster review(22:23) The impact of this process(25:15) Different types of evaluations(29:30) LLM-as-a-Judge(33:58) Improving prompts and system instructions(38:15) Analyzing agent workflows(40:38) Hamel’s personal AI tools and workflows(48:02) Lighting round and final thoughts—Tools referenced:• Claude: https://claude.ai/• Braintrust: https://www.braintrust.dev/docs/start• Phoenix: https://phoenix.arize.com/• AI Studio: https://aistudio.google.com/• ChatGPT: https://chat.openai.com/• Gemini: https://gemini.google.com/—Other references:• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/10.1145/3654777.3676450• Nurture Boss: https://nurtureboss.io• Rechat: https://rechat.com/• Your AI Product Needs Evals: https://hamel.dev/blog/posts/evals/• A Field Guide to Rapidly Improving AI Products: https://hamel.dev/blog/posts/field-guide/• Creating a LLM-as-a-Judge That Drives Business Results: https://hamel.dev/blog/posts/llm-judge/• Lenny’s List on Maven: https://maven.com/lenny—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Transcription

This episode hasn't been transcribed yet

Help us prioritize this episode for transcription by upvoting it.

0 upvotes

🗳️ Sign in to Upvote

Popular episodes get transcribed faster

Other recent transcribed episodes

Transcribed and ready to explore now

13:00H | 21 DIC 2025 | Fin de Semana

01 Jan 1970

Fin de Semana

10:00H | 21 DIC 2025 | Fin de Semana

01 Jan 1970

Fin de Semana

12:00H | 20 DIC 2025 | Fin de Semana

01 Jan 1970

Fin de Semana

2ª PARTE | 06 ENE 2026 | EL PARTIDAZO DE COPE

01 Jan 1970

El Partidazo de COPE

3ª PARTE | 22 ENE 2026 | EL PARTIDAZO DE COPE

01 Jan 1970

El Partidazo de COPE

3ª PARTE | 04 MAR 2026 | EL PARTIDAZO DE COPE

01 Jan 1970

El Partidazo de COPE

Comments

There are no comments yet.

Please log in to write the first comment.

Report any issue

How I AI

Evals, error analysis, and better prompts: A systematic approach to improving your AI products | Hamel Husain (ML engineer)

This episode hasn't been transcribed yet

Other recent transcribed episodes

13:00H | 21 DIC 2025 | Fin de Semana

10:00H | 21 DIC 2025 | Fin de Semana

12:00H | 20 DIC 2025 | Fin de Semana

2ª PARTE | 06 ENE 2026 | EL PARTIDAZO DE COPE

3ª PARTE | 22 ENE 2026 | EL PARTIDAZO DE COPE

3ª PARTE | 04 MAR 2026 | EL PARTIDAZO DE COPE

Sign in to Audioscrape

Share this moment