Most AI Pilots Fail on Integration, Not on Model Quality

The demo works. That is the problem. A pilot that produces a convincing output in a notebook has answered the easiest question in the project, and the team reads that answer as evidence the hard part is done. It is not. The hard part is everything the notebook did not have to do: authenticate, handle the record that has a null where a string was expected, decide what to do at 2am when the provider returns a 529, and write a result somewhere a human will actually see it.

Pilots that stall almost never stall on accuracy. They stall because nobody could name the system of record the output was supposed to land in.

The demo answers a question nobody was asking

A pilot typically proves that a model can perform a task on a curated sample. Someone picked fifty documents, ran them through, and the results looked good. What that establishes is capability, which was rarely in doubt for the kind of general tasks most organisations want: summarising, extracting, classifying, drafting.

What it does not establish is any of the following:

1. Whether the fifty documents resembled the actual distribution, including the scanned faxes and the ones in the wrong language.
2. What the system does when the model returns something malformed, which it will, at some rate.
3. Who is accountable for a wrong output that reached a customer.
4. Whether the output writes back to the CRM, the ticketing system, or a spreadsheet nobody opens.
5. What the thing costs at real volume rather than at demo volume.

None of these are model problems. All of them are integration and ownership problems, and they are the ones that consume the months between a working pilot and a working system. Our note on how long an AI integration realistically takes covers the shape of that gap in more detail.

What the projects that shipped had in common

Across projects that reach production, a few things recur. They are unglamorous.

Trait	Pilots that stalled	Pilots that shipped
Output destination	Decided later	Named on day one, with a field to write to
Failure handling	Assumed rare	Designed first, with a human queue
Data access	Requested during the pilot	Approved before the pilot started
Success measure	"It looks good"	A number tied to an existing process metric
Owner	The project team	A named person in the operating team
Scope	The whole workflow	One decision inside the workflow

The last row matters more than the rest. A pilot aimed at replacing an entire process has to be right about everything before it is worth anything. A pilot aimed at one decision inside that process, with the human still holding the surrounding steps, is useful at partial accuracy. It can go live at seventy percent correct because the remaining thirty percent falls back to what people were doing anyway.

The second row is the one teams find easiest to skip. If your design has no answer for a wrong output, the first wrong output stops the project. Building the review queue before the model is good is counter-intuitive and it is what allows the model to be merely good enough.

The permission question kills more pilots than the model does

A pattern worth naming: a technically successful pilot presented to a governance forum, followed by a question about where the data goes, followed by six months of nothing. The project was not rejected. It was never approved, which looks the same from the outside and is harder to escalate.

This happens because the data question is treated as a deployment detail rather than an architectural input. It is not a detail. Whether customer records can leave the network determines whether you are building against a hosted API or running something inside your own boundary, and that decision changes the cost model, the latency profile, and the engineering effort substantially. The trade-offs are laid out in our comparison of hosted APIs against self-hosted models, and the specific question of what actually happens to company data sent to AI providers is worth settling with your legal and security people before any code is written, not after.

Settle it early and you might build something slower and less capable. You will also build something that can be deployed.

Testing is the thing that gets skipped and then blocks the launch

Pilots are evaluated by looking at the output. Production systems cannot be, because there is too much output and no one is looking. The move from one to the other requires an evaluation set: a fixed collection of inputs with known-acceptable outputs, run on every change, with a pass threshold agreed in advance.

Teams resist this because it feels like overhead on something that already works, and because building the set requires someone with domain knowledge to spend real hours labelling examples. That resistance is why the system cannot be changed later. Without an evaluation set, every prompt tweak is a gamble, no one can say whether a model version upgrade improved or degraded behaviour, and the safest action is to change nothing. Systems that cannot be changed do not survive contact with a shifting business. The practical mechanics of building one are in our guide to testing an AI integration.

There is a related trap in legacy environments. A pilot run against a database export is not evidence that the same data can be read live, and the difference between the two is often a quarter of engineering work in a system with no API. That gap is worth measuring before the pilot, not after.

What to do next

If you have a pilot that works and has not shipped, do not improve the model. Answer six questions instead, in writing, with names attached: which field in which system receives the output; who reviews the ones the system is unsure about; what the acceptable error rate is and against which baseline; whether the data may leave your network and who approved that; what it costs at full monthly volume; and who in the operating team owns it once the project team leaves.

If you cannot answer them, the pilot is not close to production regardless of how it performs. If you can, the remaining work is ordinary software engineering, which is a much better position to be in than a stalled proof of concept.

For a view of how this sequencing works in practice, our AI integration services page sets out the stages we work through and where the decision points fall.