AI PM Crash Course 2026: Prototyping → Observability → Evals

episode

Previously titled “AI PM Crash Course: Prototyping → Observability → Evals + Prompt Engineering vs RAG vs Fine-Tuning” — renamed by the publisher on Aug 3, 2026

The Growth Podcast not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

recent transcribed audio

#502 – Psychiatry, Insane Asylums, Mental Illness, ECT, Lobotomies, Freud & Jung 3h 43m · 17 Sep 2026 · Lex Fridman Podcast

Your partner wants profit. You want freedom. Now what? 22 min · 16 Sep 2026 · The Optometry Success Podcast

Board of County Supervisors Meeting - Work Session - Sep 15, 2026 3h 17m · 15 Sep 2026 · Prince William County Board of Supervisors

Finance and Budget Committee - Sep 15, 2026 1h 43m · 15 Sep 2026 · Prince William County Finance and Budget Committee

Sept. 15, 2026 Board of Supervisors Meeting 8h 46m · 15 Sep 2026 · Fairfax County Board of Supervisors

The Art of True Happiness | Dr. Arthur Brooks 2h 49m · 14 Sep 2026 · Huberman Lab

Report any issue

Source

Recorded15 Jun 2025

Publisherapi.substack.com ↗

Handlerec:3786371

Show notes

Every PM has to build AI features these days. And with that means a completely new skill set:- AI prototyping- Observability, Akin to Telemetry- AI Evals: The New PRD for AI PMs- RAG v Fine-Tuning v Prompt Engineering- Working with AI EngineersSo, in today’s episode, I bring you a 2-hour crash course into becoming a better AI PM.I’ve teamed up with Aman Khan.When it comes to people creating AI PM content, Aman Khan is amongst the most insightful and informed. And that's because he's been an AI PM since 2019:- He worked at Cruise on self-driving cars. - He's worked with Spotify on their AI systems. - And now he works at Arize, one of the leading observability and evals companies.----Brought to you by:Miro: The innovation workspace is your team’s new canvasJira Product Discovery: Plan with purpose, ship with confidenceMaven: Get $100 off Aman’s course with my code ‘AAKASHxMAVEN’Amplitude: Test out the #1 product analytics and replay tool in the market----Timestamps:Can Anyone Become AIPM? - 0:005 AIPM Skills Overview - 5:52Skill 1: AI Prototyping - 6:31Ad: Miro - 13:35Ad: Atlassian - 14:50Building Trip Planner Agent - 15:27Ad: Maven - 29:46Ad: Amplitude - 30:40Skill 2: Observability - 50:34Skill 3: Evals - 1:10:10RAG vs Fine-Tuning vs Prompt Engineering - 1:29:54Bolt Teardown - 1:30:32Skill 5: Working With Engineers - 1:43:24Don't Make These Mistakes - 1:48:332 Hours Weekly Plan - 1:53:55AIPM Jobs Exist - 1:57:45Aman's Resources - 2:00:48Outro - 2:04:00----Key Takeaways:1. Cursor beats Bolt for serious AI PMs. While Bolt is great for quick mockups, Cursor gives you the control you need to build real agent systems and understand what's happening under the hood.2. Observability comes before evals. Just like regular products need telemetry for analytics, AI products need traces for evals. Point Cursor to documentation and it adds what you need.3. Vibe coding doesn't scale. Looking at outputs and deciding if they "feel good" works for prototypes, but not production. You need systematic evals to measure what "good" actually means.4. Most PMs fine-tune too early. Aman showed a prompt outperforming a fine-tuned model. Start with prompting (95% of results), add RAG for external data, only fine-tune for cost/speed.5. Your evals need evals. When your LLM judge marks outputs as "friendly" while your human labels say "robotic," that mismatch tells you exactly where to improve your system.6. Use text labels, not numbers. LLMs understand "friendly vs robotic" better than 1-5 scales. They're trained on language, not mathematics.7. AI engineers want data, not docs. Stop sending Google Docs with requirements. They want you labeling datasets and defining success through evals.8. Bolt is just a really good prompt. Aman tore down Bolt's architecture - it's system prompts + tool calling + code generation. The "magic" isn't magic.9. Side projects are your interview hack. When Aman asks "What are you building?" he can immediately gauge curiosity, initiative, and hands-on experience.10. Don't automate yourself too early. Use AI as a second brain for analysis, but don't try to automate your entire job. Learn to work with reasoning models to push your thinking.----Check it out on Apple, Spotify, or YouTube.----Where to Find Aman:LinkedIn: Aman KhanX: Aman KhanSubstack: aiproductplaybook.comCompany: ArizeCourse: The AI PM Playbook----Related Podcasts:Tutorial of Top 5 AI Prototyping ToolsComplete Course: AI Product ManagementWe Built an AI Agent to Automate PM in 73 mins (ZERO CODING)We Built an AI Product Manager in 58 mins (Claude, ChatGPT, Loom + Notion AI)We Built an AI Employee in 62 mins (Cursor, ChatGPT, Gibson, Crew AI)----Up NextI hope you enjoyed the last episode with Tom Occhino (where we gave an in-depth v0 tutorial). Up next, we have episodes with:Dan Olsen - Author, Lean Product PlaybookJohn Beckmann - Head of Events + Webinars, ZoomTanguy Crusson - Head of Product, Jira Product DiscoveryFinally, check out my latest deep dive if you haven’t yet: The Playbook to Land Your First PM Job.----If you want to advertise, email productgrowthppp at gmail. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.