In partnership with
Separating AI Hype from Real Business Value.
In late 2024, I was working with a company that had a sales problem, yet some leaders focused on building AI features.
Interestingly, the core problem of the company — revenue — was not front and center for the leadership team. AI took center stage, and most important engineering resources got pulled from the core product to build ‘innovative AI projects’.
They had read about companies transforming their products with AI. They believed their project would look like those stories.
Nobody asked: what is happening to companies like ours that did this?
That is survivorship bias. And it is everywhere in AI right now.
In today's newsletter:
Why we only see AI's winners and never its graveyards
The real numbers behind AI project failure (they are worse than you think)
How to evaluate AI before your team builds anything
ChatGPT Is the Bill Gates of AI
Survivorship bias is when we study only what survived and overlook what didn't.
In Outliers, Malcolm Gladwell writes about Steve Jobs, Bill Gates, and Mark Zuckerberg, all dropped out of university. Someone looks at that and concludes: dropping out is a key to success in tech. What is missing? The thousands of dropouts who went nowhere. Nobody writes articles about them.
ChatGPT is the Bill Gates of AI. The dropout who became a billionaire.
But there are dropouts everywhere. McDonald's tested an AI drive-through in partnership with IBM. In one incident, the AI ordered a customer 18,000 cups of water. The program was quietly shut down.
Projects like these are far more common than the billion-dollar success stories,
They are just less likely to be discussed.
Why 80% of AI Projects Never Reach Production
This data can help us avoid survivorship bias. When an AI project is imagined, most teams reference the success stories. These numbers show what happens when we look at the full picture instead.
US businesses have invested between $35 and $40 billion in generative AI. Very little to show for it.
These are the numbers that don't make it into the conversation. And this is why the better question is not "how do we add AI?" but
"what burden can AI actually remove?"
Problem First, AI Later
The company in my opening had a revenue problem. Users were churning. The real question was:
how can we solve our users' problems so they stay?
AI could have helped: Understanding where users were dropping off, reducing friction in onboarding, surfacing patterns in churn data. Instead, The team built an AI feature that looked impressive in a demo but had nothing to do with the user's actual problem.
According to my experience, this is where the user experience conversation gets missed.
When FOMO drives the AI decision, nobody starts with the customer experience. Nobody maps the user's real friction.
And the user experience suffers — inaccurate outputs, unpredictable results, features that add complexity without solving anything. When the user experience gets worse, the company's original problem gets worse too.
Start with the customer. Find their burden.
Think about Nadia.
She is a product manager and she is the user of your product. Her week splits into two kinds of time.
Goal Time
Tool Time
Goal Time is customer conversations, discovery, trade-off decisions — the work Nadia came to do.
Tool Time is the friction: status updates, formatting PRDs, updating Jira, pulling dashboard data, searching Confluence.
When AI targets Tool Time, Goal Time expands.
And something else appears: time back.
Time Nadia gets to reclaim for her life
Why This Matters for Your Next Project
In 2024, I did not ask these questions. The team I worked with started with excitement, not with the customer's real problem. We saw what AI could do in other companies. We did not look at what it failed to do in companies like ours.
Three years into AI, we have enough data to make better decisions. These numbers are not reasons to avoid AI. They are reasons to start with your company's objective the user's real problem and work backward to the technology.
The companies that will get value from AI are the ones that resist the FOMO and ask: what burden can we actually remove? What friction is costing us users? And is the technology ready to solve that specific problem today?
Problem first, AI later.
According to my experience, it is the only approach that works.
Key Takeaway
Problem first, AI later. Steve Jobs said it in 1997. The principle has not changed yet we continue making the same mistakes.
Survivorship bias fuels the FOMO. We see AI's wins. The 80% that fail before production and the 95% that deliver no measurable impact disappear from the conversation.
If AI does not improve how the user experiences the product, it is not solving the problem. It is adding to it.
P.S. The $35 to $40 billion already spent on generative AI is not a success story. It is a reference class. Study it before you add to it. If you missed my last article on Reference Class Forecasting, you can read it here.
AI Tools I'm Using This Week
💬 Claude — strong momentum, but now hitting limits as growth raises questions around consistency and reliability
🧠 ChatGPT — after stepping away for a month while testing Claude, I’m back. It still performs better for clear English variations and structured writing
⚙️ Open Claw — testing it this week for local automation on my system. Early focus is workflow control, not just outputs
I’ll share what holds up and what breaks in next week’s update
1,000+ Proven ChatGPT Prompts That Help You Work 10X Faster
ChatGPT is insanely powerful.
But most people waste 90% of its potential by using it like Google.
These 1,000+ proven ChatGPT prompts fix that and help you work 10X faster.
Sign up for Superhuman AI and get:
1,000+ ready-to-use prompts to solve problems in minutes instead of hours—tested & used by 1M+ professionals
Superhuman AI newsletter (3 min daily) so you keep learning new AI tools & tutorials to stay ahead in your career—the prompts are just the beginning