Rob Wiblin

Thank you too.

If you were following technology news in August last year, you almost certainly heard about this MIT study showing that 95% of generative AI pilots at companies were failing.

This result was big enough to contribute to a NASDAQ sell-off.

People worked hard to come up with sophisticated explanations for how something this crazy could be true.

And it was repeated by Forbes, Axios, The Hill, the Harvard Business Review, and dozens of others, becoming a staple of elite opinion and one of the most enduring and widely cited statistics in the AI is overhyped backlash.

The problem?

The study behind these headlines is incredibly weak.

Worse than you could imagine.

And that headline is also a completely incorrect description of what it found, even taking the study entirely on its own terms.

The story behind this study will demonstrate that whenever you see a juicy headline, even one with an attractive conclusion, and even one rewarding to come from MIT, it might just be complete nonsense.

The most important thing to know is that this report did not show that 95% of generative AI pilots at companies are failing, as almost all journalists claimed.

Rather, the report found that of all the organisations surveyed, 60% had investigated custom enterprise AI tools, 20% had gotten to the point of actually doing some pilot project with them, and 5% of the total had gone on to successfully deploy those tools in production.

80% of companies simply never piloted any custom task-specific generative AI.

Saying that 95% of them were failing is like saying 95% of Tinder users have failing marriages when 80% of the people you're talking about have never even gone on a date in the first place.

Moreover, according to their own survey, the primary reason why pilots didn't progress to deployment wasn't that they were going badly, but just the very familiar and generic organizational unwillingness to adopt new tools.

Now, the media is definitely at fault for putting a completely wrong number in their headlines, but the report also makes this mistake about its own graph, referring to a 95% failure rate for enterprise AI solutions.

Wrong.

These results actually show that among the 20% who did in fact pilot a custom AI tool, about 25% or a quarter were successful.

To qualify as a success, an AI application had to show a marked and sustained productivity or profit and loss impact within six months.

They don't define marked or sustained, but a marked and sustained improvement in profitability or productivity within six months is obviously a high bar for any new project to clear.