Free executive brief · Agent Playbooks

The "95% of AI pilots fail" number in your board deck is wrong.

It comes from a preliminary, non-peer-reviewed paper built on 52 interviews. The methodology quoted in the press contradicts the methodology printed in the PDF. And the figure everyone repeats is not the figure the paper reports.

Published July 2026 by Anodize Labs. Every source below was checked against a live primary document on 26 July 2026, and the date is printed next to each one. Nothing on this page is gated, there is no popup, and you do not need to give us an email address to read any of it.

---

1. The claim you have probably repeated

Some version of this sentence has been in front of nearly every executive in the last year: ninety-five percent of AI pilots fail. It appears in board decks, vendor pitches, consultancy proposals and internal memos arguing both for and against spending more money. It is the closest thing enterprise AI has to a shared fact.

It is worth knowing exactly what produced it, because the answer changes what you should do with it.

2. Where the number actually came from

The source is The GenAI Divide: State of AI in Business 2025, by Aditya Challapally, Chris Pease, Ramesh Raskar and Pradyumna Chari, published under Project NANDA at the MIT Media Lab in July 2025 and picked up widely through Fortune in mid-August 2025. Its own notes page states the method:

Now compare that with the description that circulated. Secondhand write-ups characterised the study as 150 interviews with leaders, a survey of 350 employees, and an analysis of 300 public AI deployments. The paper says 52 and 153. The methodology most people can recite for this statistic is not the methodology of the paper the statistic comes from.

What the 95% actually measures

The report describes a funnel for custom or vendor-sold, task-specific generative AI tools: 60% of organisations evaluated one, 20% piloted, and 5% reached production. That is where 95% comes from. It is a production rate for a narrow category of build, measured six months out.

The same report measures general-purpose tools such as ChatGPT and Copilot separately, and the numbers are almost inverted: 80% investigated, 50% piloted, 40% implemented. That is a pilot-to-implementation rate of roughly 83%.

Five problems, four of them conceded by the authors

1. Not peer reviewed, and self-labelled preliminary. The cover page reads "Preliminary Findings from AI Implementation Research from Project NANDA." The listed reviewer is also a listed author. There is no journal and no replication package.
2. Small and non-random. 52 interviews and 153 conference attendees. The report concedes "Selection bias possible in organizations willing to participate in AI research."
3. A subjective and short success bar. Success is partly coded from whether "users or executives have remarked" on impact, measured six months after the pilot. The report itself warns that the "Six-month observation period may be insufficient... potentially understating success rates."
4. A structural conflict of interest. NANDA builds agentic and memory interoperability infrastructure, and the report's prescribed remedy is agentic systems plus those protocols. The diagnosis deserves more weight than the prescription.
5. The circulating methodology contradicts the document. This one is not conceded, because the authors did not write it. It was added in transmission.

The report also states its own limitation plainly: "These figures are directionally accurate based on individual interviews rather than official company reporting. Sample sizes vary by category, and success definitions may differ across organizations."

Lead author Challapally has since publicly softened the framing, saying that "Some large companies' pilots and younger startups are really excelling with generative AI."

3. What the evidence actually supports

Here is the defensible version of the sentence, and it is still a useful finding:

Cross-check that against the larger surveys and a range appears rather than a number.

The honest range for "pilots that produce measurable value" is roughly 5% to 40%, and where you land inside it is decided entirely by what you count. None of these are independent of each other in the way they are usually presented, either. Stanford HAI's widely quoted adoption percentages are imported from the McKinsey survey, so citing both for adoption is citing one survey twice.

4. The finding that retires your current ROI case

Almost every AI business case in circulation rests on time saved, and almost every estimate of time saved comes from asking people how much time they saved. There is now a randomised controlled trial that measures both at once.

METR, a nonprofit evaluation organisation with no vendor funding, ran 16 experienced open-source developers across 246 randomised tasks in mature repositories they had worked in for years. From the paper:

Expected 24% faster. Believed 20% faster afterwards. Measured 19% slower. The gap between perception and measurement is the finding, and it is roughly 39 percentage points wide.

State the caveats, because they matter

n=16 is small and the confidence interval is wide. The study deliberately chose the least favourable case for AI: experts working in codebases they know deeply. It is not evidence that AI slows everyone down, and anyone presenting it that way is making the same mistake in the opposite direction. What it does establish is narrower and more damaging to your reporting: practitioners cannot introspect their own productivity change, and they get the sign wrong.

The rest of the causal evidence

Surveys measure opinions. These measure output. Taken together they point somewhere quite specific.

The pattern is consistent and it is the opposite of how most rollouts are targeted. Every rigorous study finds gains concentrated in less-experienced staff on well-specified tasks, and neutral to negative effects for experts doing complex work in systems they already know. Most enterprises pilot with their senior people, because that is where the political capital sits. The evidence says pilot where the skill floor is.

5. The governance dates in your policy document are probably stale

Two changes landed recently enough that most published guidance, including several popular compliance trackers, has not caught up.

The EU AI Act's high-risk deadlines moved

Regulation (EU) 2024/1689 entered into force on 1 August 2024 with a staged timeline. The Digital Omnibus on AI, procedure 2025/0359(COD), was proposed on 19 November 2025 and moved the two high-risk dates. It is adopted, not merely proposed: Parliament adopted it in plenary on 16 June 2026 (423 for, 57 against, 174 abstentions), the Council adopted it on 29 June 2026, and the final act was signed on 8 July 2026.

One honest caveat on this one. The Commission's regulatory framework page describes the omnibus as having entered into force in July 2026, while the Legislative Observatory still showed the file as completed and awaiting publication in the Official Journal when we checked on 26 July 2026. The substance is settled; the final Official Journal citation and the precise entry-into-force date are not yet pinned down, so do not put a citation string in a policy document until they are. We have also seen no confirmation of whether the Article 4 AI literacy duty was altered, so treat it as binding.

Two things are worth separating from the delay, because they are in force today and they catch ordinary employers with no AI product of their own:

• Article 5(1)(f) prohibits AI used to infer emotions of a natural person in the workplace, except for medical or safety reasons. Emotion or sentiment analytics pointed at your own staff is not a 2027 problem. It has been prohibited since February 2025.
• Article 4 requires providers and deployers to ensure "a sufficient level of AI literacy" among staff and others operating AI systems on their behalf, in force since February 2025.
• Article 2 reaches a company with no EU establishment where "the output produced by the AI system is used in the Union." A US-only headcount does not settle the question.
• Article 99(6) applies the lower of the percentage or the fixed amount for small and medium enterprises and startups. This clause is directly relevant to a company of 50 to 5,000 people and is almost never mentioned in the popular coverage.

Colorado's AI Act was repealed before it took effect

Colorado SB 24-205, signed 17 May 2024, was the first comprehensive US state AI law and became the template a great deal of governance advice was written against. It never applied to anyone. Its effective date was pushed from 1 February 2026 to 30 June 2026 by SB 25B-004 in August 2025, and it was then repealed and reenacted as SB 26-189, the Automated Decision-Making Technology Act, signed 14 May 2026.

The new act's statutory effective date is 12 August 2026, with substantive developer and deployer obligations running to 1 January 2027. Enforcement is by the Attorney General only, with no private right of action, penalties up to $20,000 per violation and a 60-day cure period until 1 January 2030.

If your AI governance programme was built to satisfy the original Colorado act, roughly half of what it obliges you to do is now voluntary in Colorado. That may still be the right thing to do. It is no longer the thing the statute requires, and knowing the difference is the point.

6. What to do differently on Monday

Five changes, each of which follows directly from something above and none of which requires a budget.

1. Ask for the denominator, out loud, every time. When a failure rate or a success rate is presented, the question is "counted against what population, over what window, with success defined how?" On the 95% figure alone, that question moves the answer between 5% and 83% using the same source document.
2. Stop accepting self-reported hours saved as evidence of anything. Treat it as a measure of enthusiasm, which is worth knowing and is not worth booking. Where a business case rests on time saved, ask what independent measurement would falsify it, and whether that measurement is being taken.
3. Establish a baseline before the pilot, not after it. Capture how the work happens today: time, cost and error rate, across several cycles rather than the first attempt. A pilot without a pre-agreed baseline cannot produce a result, only an impression.
4. Pilot where the skill floor is. Every rigorous study puts the gains in less-experienced staff on well-specified work. Starting with your most senior people is the politically obvious move and the empirically worst one.
5. Date-stamp your governance document and re-check the two live obligations. Put a verification date on every regulatory claim in it. Then check two things that apply today regardless of the delayed high-risk deadlines: whether anything in your stack infers employee emotion or sentiment, and whether you have anything that would satisfy an AI literacy duty.

One further note on the projections you will be shown. The most quoted enterprise AI success story of 2024, Klarna's customer service automation, was published as "projected to deliver $40 million in profit improvement." Projected, not realised, and the company later walked back parts of the automation. Never book a projection in a business case without the measurement plan that would falsify it.

Sources

Everything above traces to a document you can open. Each was checked against a live primary source on 26 July 2026. Where a citation could not be resolved to a primary document, that is said at the point of use rather than hidden here.

The 95% claim

• The GenAI Divide: State of AI in Business 2025, Challapally, Pease, Raskar and Chari, Project NANDA, MIT Media Lab, July 2025. Working PDF mirror. Verified 26 July 2026.
• Paul Roetzer, Marketing AI Institute, on what the 95% definition excludes. marketingaiinstitute.com. Verified 26 July 2026.

Base rates and market surveys

• McKinsey, The state of AI in 2025: Agents, innovation, and transformation, November 2025, n=1,993 in 105 nations. mckinsey.com. Verified 26 July 2026.
• IBM Institute for Business Value, 2025 CEO Study, 6 May 2025, n=2,000 CEOs. ibm.com. Verified 26 July 2026.
• BCG, The Widening AI Value Gap, 30 September 2025. BCG does not publish the respondent count or sampling frame on the public page. bcg.com. Verified 26 July 2026.
• Deloitte, State of AI in the Enterprise, 2026 edition, n=3,235 senior leaders in 24 countries. deloitte.com. Verified 26 July 2026.
• Menlo Ventures, 2025: The State of Generative AI in the Enterprise, 9 December 2025, n=495. menlovc.com. Verified 26 July 2026.
• Stanford HAI, 2026 AI Index Report, April 2026. Its adoption percentages are imported from the McKinsey survey and are not independent of it. hai.stanford.edu. Verified 26 July 2026.

The causal evidence

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 2025. arXiv:2507.09089 and metr.org. Verified 26 July 2026.
• Dell'Acqua et al., Navigating the Jagged Technological Frontier, HBS Working Paper 24-013 (2023), published in Organization Science. doi 10.1287/orsc.2025.21838. Verified 26 July 2026.
• Brynjolfsson, Li & Raymond, Generative AI at Work, Quarterly Journal of Economics 140(2):889, 2025. academic.oup.com. Verified 26 July 2026.
• Cui, Demirer, Jaffe, Musolff, Peng & Salz, The Effects of Generative AI on High-Skilled Work, Management Science, June 2025, pre-registered AEARCTR-0014530. Author copy. Verified 26 July 2026.
• Noy & Zhang, Science 381:187-192, July 2023. doi 10.1126/science.adh2586. Verified 26 July 2026.
• Peng, Kalliamvakou, Cihon & Demirer, February 2023. arXiv:2302.06590. Verified 26 July 2026.

Regulation

• Regulation (EU) 2024/1689 of 13 June 2024, published in the Official Journal on 12 July 2024, in force from 1 August 2024. Article and Annex references above are to the regulation text. Verified 26 July 2026.
• Digital Omnibus on AI, Commission proposal COM(2025) 836 of 19 November 2025, procedure 2025/0359(COD). Parliament plenary adoption 16 June 2026 (T10-0198/2026), Council adoption 29 June 2026, final act signed 8 July 2026. Tracked through the European Parliament Legislative Observatory and Legislative Train. Final Official Journal citation still unresolved as of 26 July 2026.
• Colorado SB 24-205 (signed 17 May 2024), SB 25B-004 (28 August 2025), and SB 26-189, the Automated Decision-Making Technology Act (signed 14 May 2026). Verified 26 July 2026.
• Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence, 11 December 2025, 90 FR 58499. Verified 26 July 2026.

The Klarna example

• OpenAI, AI in the Enterprise: Lessons from seven frontier companies. cdn.openai.com. The $40 million figure appears there as a projection. Verified 26 July 2026.

---

If this was useful

This brief is three corrections. It is not a sales page with a chapter attached, and if you close the tab here you have the part that changes what you do on Monday.

We also publish The Agent Playbook for Executive Leaders, which is built the same way: named sources with sample sizes, dates and sponsors, and a plain statement wherever a widely repeated number is weak. It covers where value actually shows up by function, build versus buy versus adopt, org design, budget and unit economics with dated vendor prices, measurement, and rollout. It costs $199 once, it is a PDF, and there is a 60-day refund with no questions asked. It is genuinely not for you if you want a certification, a cohort, or a hands-on prompting course.

Agent Playbooks is published by Anodize Labs, an independent shop in Spokane, Washington. We are not affiliated with, endorsed by or sponsored by MIT, METR, Anthropic, OpenAI, or any organisation named on this page, and no one paid for a mention. Contact details and business information.