Back in the day, Spotify built a custom A/B testing tool called ABBA. It was great. The platform enabled lots of teams to try out lots of ideas for new features to see what worked and what didn’t. With ABBA, we went from doing tens of experiments to hundreds of experiments. But we didn’t just learn what color button users liked better: the more tests we ran, the more we learned about our testing methods, including the limitations of ABBA itself — which eventually led us to a new, better way to test. Here’s the story of ABBA, our very first experimentation platform, and the lessons we learned about doing product experimentation at scale. Host Dave Zolotusky talks with Mark Grey, a senior staff engineer and 10-year Spotify veteran. They discuss Spotify’s earliest efforts at product testing, our early infrastructure for data and data processing (using Hive and Hadoop), how migrating to the cloud unlocked more processing power (and more testing), the difference between using tests to design the color of a button and using tests to inform the very next user interaction via machine learning, feature flags and holdout groups, all the things we learned about conducting scientifically sound experiments, how we built a culture of experimentation among our software development teams, and what finally drove us to sunset ABBA and build its successor: a bigger, better internal experimentation platform. Plus, progress bars and lightsabers. Read more about ABBA and how we do product experimentation at Spotify: Spotify’s New Experimentation Platform (Part 1) Spotify’s New Experimentation Platform (Part 2) Search Journey Towards Better Experimentation Practices Spotify’s New Experimentation Coordination Strategy Choosing a Sequential Testing Framework — Comparisons and Discussions Comparing Quantiles at Scale in Online A/B-Testing Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com You should follow us on Twitter @SpotifyEng and on LinkedIn!

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related