In this episode, Richard Bradshaw and Vernon discuss the relevance and application of the six principles of automation in testing in the context of AI advancements. They explore how these principles hold up in 2026, the challenges faced in automation, and the future of testing strategies.00:00 - Intro01:47 - Welcome (Richard is not at home 👀)02:07 - Ramadan, cooking without tasting, and plastic teeth 🦷04:01 - Today's topic: revisiting the AiT principles ahead of a keynote04:58 - What is Automation in Testing (AiT)?06:49 - Principle 1: Supporting Testing over Replicating Testing07:01 - Vernon's take: testing is a performance, not a click sequence08:22 - What the industry promised vs what automation actually does08:49 - The serendipity you lose when a human isn't testing09:59 - Agentic testing: observing more, but still not replicating humans10:56 - The danger of anthropomorphising AI output12:10 - LLMs always give an answer — and that's the problem13:03 - Principle 2: Testability over Automatability13:14 - Vernon's take: narrow vs broad — operate, control, observe14:38 - Making apps automatable for the robots but not the humans15:37 - The shiniest framework in a broken testing context16:40 - If it's testable, it's probably automatable — but not vice versa16:55 - Automation strategy vs testing strategy: when they compete, everyone loses17:46 - The problem has always been testing, not automation19:57 - Principle 3: Testing Expertise over Coding Expertise20:18 - Vernon's take: testing expertise lets you leverage the tools21:47 - The spoonfed tests problem: great at automating, lost without guidance22:36 - The "code school" era: everyone told to learn to code22:51 - Coding agents have changed the maths on this26:01 - The new nuance: test design and framework knowledge over writing the code28:44 - Evaluating code is a testing problem — and LLMs can help you do it30:43 - Are agents as good as a junior developer?31:42 - Outcome Engineering (O16G) and the race to write the AI principles32:13 - Simon Wardley: we're in the wild west again33:22 - Principle 4: Problems over Tools33:29 - Vernon's take: the hammer and the nail34:07 - Don't let your problems be shaped by the framework you have34:36 - New automation opportunities beyond testing: PRs, logs, story review35:30 - Principle 5: Risk over Coverage36:12 - Vernon's take: 100% coverage ≠ 100% risk coverage38:00 - The one test case, one automated test fallacy39:04 - Where in the system is the risk? Do you even know your layers?39:49 - Probabilistic vs non-deterministic: refining the language around AI40:53 - Coverage as intentional vs coverage as a number someone picked once43:15 - Principle 6: Observability over Understanding43:24 - Vernon's take: just-in-time understanding vs reading everything upfront44:12 - What the principle was actually about: making automation results observable47:00 - Does this principle belong in testing, or has it grown into quality?49:00 - So... what's missing?50:00 - The four pillars: Strategy, Creation, Usage, and Education57:05 - Automation in Quality: the bigger opportunity01:01:00 - Wrap up + Vern's Lead Dev panelLinks to stuff we mentioned during the pod:04:00 - Automation in Testing (AiT)The principles live at automationintesting.comAiT was co-created by Richard Bradshaw and Mark Winteringham04:00 - Test Automation DaysThe conference where Richard is giving his keynote — testautomationdays.com24:48 - James ThomasThe "kid in a candy shop" himself — James's blog and LinkedIn31:42 - Outcome Engineering (016G)The article Richard shared before recording — worth tracking down if you're interested in where agentic development practices are heading32:13 - Simon WardleyIf you're not following Simon Wardley, please follow Simon Wardley! His work on Wardley Maps and situational awareness in strategy is essential readingSimon's LinkedIn43:30 - Abby BangserVern's go-to person for all things observability. Abby's LinkedIn46:04 - Noah SusmanAs it turns out, the quote Vern's referencing: advanced monitoring as "indistinguishable from testing" was not by Noah! It was Ed Keyes at GTAC 2007.Noah's blog and LinkedIn59:30 - Angie JonesVern's been reading Angie's work on testing AI-enabled applications here and here.Angie's website and LinkedIn01:01:30 - The Lead Dev panel Vernon will be part of"How to Measure the Business Impact of AI" — happening 25th February, free to sign up01:02:00 - Richard's Selenium Conf talk"Redefining Test Automation" — the talk that the Test Automation Days keynote is shaping up to be a spiritual successor to.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related