Most AI pilots we're asked to rescue share a story. A small team builds an impressive demo in a few weeks. Leadership is excited. Then the project spends six months in a state that's neither live nor dead, until someone quietly stops paying for the API key.
The model is rarely the problem. The gaps are organizational and architectural, and they're predictable enough that we now check for all four before agreeing to build anything.
1. There's no evaluation set
A demo is judged by whether it impresses the room. Production is judged by whether it's right often enough, on the inputs you actually get. Without a fixed set of real examples and agreed "good" answers, every change is an argument about vibes.
What we do: before any build, we collect 100–300 real examples with the people who do the work today, and agree on how each will be scored. That set becomes the regression test for every prompt, model or retrieval change after it.
2. Nobody owns it
Pilots are often sponsored by innovation budgets, not by the team whose process changes. When the pilot ends, there's no one whose job it is to adopt it.
What we do: we name a business owner — a person, not a department — who signs off on the evaluation set and on go-live. If we can't find one, that's the finding.
3. There's no path to the data
The demo ran on an export someone emailed over. Production needs a governed, repeatable way to read from the system of record, with the right permissions, logged.
What we do: map the data path in week one — source, access method, refresh frequency, and who approves access. This often turns into a small data engineering project, and it's better to know that up front.
4. There's no cost ceiling
Usage-based pricing is easy to ignore at pilot scale and alarming at production scale. Finance kills projects that can't answer "what will this cost per month at full volume?"
What we do: model the unit cost per task (tokens, retrieval, human review time) against the cost of doing it today. We set a hard monthly ceiling with alerts, and design fallbacks for when it's hit.
The pattern
None of these are glamorous, and none require a better model. They require treating an AI feature like any other production system: measurable, owned, connected and budgeted. Get those four in place and the model choice becomes one of the easier decisions you'll make.