The technology didn’t fail. The pilot succeeded — at being a pilot.
MIT’s NANDA initiative examined more than 300 publicly disclosed enterprise AI deployments alongside dozens of executive interviews and leadership surveys. Against an estimated $30 to $40 billion in enterprise investment, 95% of generative AI pilots produced no measurable profit-and-loss impact. Only about 5% reached production with value attached. The funnel underneath is sharper still: 60% of organizations evaluated enterprise-grade tools, 20% got as far as a pilot, and 5% went live.
The models weren’t the constraint. Researchers traced the failures to everything around them — brittle workflows, systems that couldn’t retain feedback or adapt to context, and pilots built on hand-curated data that looked nothing like what arrived in production. There’s a second finding buried in the same study that leaders tend to skip past: deployments built with specialized outside partners reached production roughly 67% of the time, while purely internal builds succeeded about a third as often.
That distinction changes what the 95% actually means. These weren’t bad bets on bad technology. They were real results, produced under controlled conditions, that evaporated the moment those conditions widened. Which is a much older problem than AI — and one that already has a name.
That gap — between a result that holds in a test and a result that holds at scale — is exactly what University of Chicago economist John List takes on in The Voltage Effect. His term for the collapse is a voltage drop: the moment an idea that worked in a small setting loses its charge as it expands. Borrowing the concept from implementation science, and pointing to evidence that somewhere between 50 and 90 percent of programs lose voltage at scale, List’s argument is that this isn’t bad luck. It’s predictable, and it’s diagnosable before the money is committed.
Consider D.A.R.E., the book’s opening case. The program was persuasive, well-funded, and politically irresistible; over twenty-four years, 43 million children across more than forty countries went through it. Study after study later found it didn’t reduce drug use. The failure wasn’t that the idea was tested and proved wrong. It’s that it was scaled to forty countries before anyone tested it properly at all.
The 95% figure has a counter-example built into the same study, and it’s worth sitting next to D.A.R.E. rather than treating the collapse as the whole story. The MIT team’s own lead researcher points to startups — some run by founders barely out of their teens — that took generative AI from zero to $20 million in annual revenue within a year. Not by building more, or building broader. By picking one specific pain point, executing it well, and partnering with tool-builders instead of trying to build everything in-house. The gap between the 95% and the 5% wasn’t effort or ambition. It was discipline about what to test, and how honestly to read the result.
Two ideas worth carrying into the boardroom: Treat a successful pilot as a hypothesis, not a verdict. The most common reason results vanish at scale is that they were never as strong as they looked — a small sample, a motivated team, and a favorable setting can manufacture a signal that was never really there. And scaling is a weakest-link problem. List identifies five specific failure points — false positives, an audience that doesn’t represent the real population, ingredients that can’t be replicated, costs that climb faster than volume, and spillover effects that only appear at size. Any one of them is enough to sink the whole thing, which means the right question before a rollout isn’t “did it work?” but “which of the five would break it?”
Read The Voltage Effect →
Best for: Executives deciding which pilots deserve real budget, not teams looking for a framework to run the pilots themselves. Reading commitment: Around five hours, accessible and example-driven rather than technical.
That shift — from trusting a promising result to checking whether it survives a wider population — is exactly what Optimizely is built to make routine rather than heroic. It’s an experimentation platform where testing is the entire business, not a feature: teams run controlled experiments against real traffic, and its statistics engine is designed specifically to stop the false-positive problem List describes, holding results to a standard that doesn’t reward peeking at the data until a winner appears. Its Opal layer adds AI agents that monitor experiments for significance, generate plain-language summaries of what actually happened, and break results down by segment — so a leader can see not just whether something worked, but for whom.
It won’t tell you which initiatives are worth testing, and it’s built for digital products and experiences rather than every kind of operational rollout. It also demands something most organizations find harder than the software: a willingness to let a favored idea fail in public, on the record. But for leaders who’d rather find out a pilot won’t scale before it becomes a line item, it makes the difference between a real result and a flattering one visible while it still costs almost nothing.
There’s also a chance to put your own thinking in front of a wider audience: this issue brings an exclusive feature placement in USA Today, arranged through TEI’s media relationships. USA Today reaches well beyond the trade press to customers, employees, and partners — the audiences that notice which companies have actually shipped something, rather than which ones announced a pilot, which is exactly the distinction this issue is about. TEI’s team handles the interview, drafting, and placement end to end.
Interested? Talk to our team to learn more about this opportunity and discuss the details. Schedule a call through our Calendly, or simply reply to this email and we’ll be happy to assist.
Take with you: Of the AI initiatives currently on your roadmap, how many were approved on a result someone actually stress-tested — and how many on a demo that went well?
The Executive Insight — 169 Madison Ave STE 11570, New York, NY, 10016
