How to choose an AI integration partner
Almost anyone can build an AI demo now, which has made demos worthless as a signal. What separates a partner worth hiring is what happens after the demo works: when the model is unsure, when the provider is slow, when the bill arrives, and when accuracy drifts six months later. These are the questions that reach that.
What should you ask before signing?
Ask about the failure paths, not the capability. Every proposal describes what the system does when it works. Very few describe what it does when it does not, and that difference is most of the gap between a pilot and something you can run.
| Ask this | Why it matters |
|---|---|
| What happens when the model is unsure? | There should be a confidence threshold and a defined escalation path. No answer means it will guess, confidently, in production. |
| What will this cost to run at ten times this volume? | Inference cost scales with usage, not seats. A build-only price is an incomplete price. |
| How will we know if accuracy drops in six months? | Model behaviour drifts as providers update and your data shifts. Without sampling and monitoring, you find out from a customer. |
| What is your evaluation set, and can we see the score? | A partner without labelled test cases is asserting quality rather than measuring it. |
| Which model, and what licence does it carry? | Some open weights are copyleft or non-commercial, and the obligation lands on you rather than on them. |
| Does our data leave our infrastructure, and to whom? | Determines whether your privacy notice and contracts already cover it. Cheaper to ask now than during a procurement review. |
| What is the baseline we are trying to beat? | If nobody measured the current process, nobody can prove the project worked. |
| Who owns the prompts, the evaluation set and the code? | Agree it before handover, not during it. |
Which answers should worry you?
"We will use the latest model." That is a purchasing decision presented as engineering. The right answer names a model and explains why it fits the task, usually including the observation that a smaller one handles most of the work more cheaply.
A proposal with no line for evaluation or monitoring. It means the plan ends at the demo, and the ongoing quality problem becomes yours to discover.
Accuracy quoted as a percentage with no dataset behind it. Ninety-five per cent of what, measured on which cases, labelled by whom? Without answers, the number is decoration.
Reluctance to discuss what the system will not do. Every honest scope has exclusions, and a partner who agrees to everything in the first meeting is deferring the disagreement to the point where it costs you money.
And a proposal that opens with replacing a platform you already run. Occasionally that is genuinely necessary. Far more often it is a larger project sold in place of a smaller one that would have worked.
How do you compare proposals that are not comparable?
Normalise three things before looking at price. Whether running costs are included or only the build. Whether evaluation and monitoring are in scope or extra. And whether the price assumes your data is clean, because a data preparation pass frequently costs more than the integration and is the most common thing quietly excluded.
Then compare on the same workflow, deliberately narrow. Ask each partner to scope exactly one process end to end rather than an AI strategy. Strategy proposals are impossible to compare and impossible to hold anyone to; a single workflow with a baseline and a target is both.
Be suspicious of the cheapest and the most expensive for the same reason: both usually indicate a different understanding of the scope than yours. Ask each to state what they think the hard part is. The answers will differ more than the prices, and that difference is the useful information.
Should you use a specialist, a generalist agency, or hire?
Hiring is right when AI is going to be a permanent part of your product rather than a project. The work does not end at launch, and someone has to own drift, cost and model updates indefinitely. If that person does not exist, an agency build eventually becomes an orphan.
A partner is right for the first one or two integrations, where the value is knowing which of the many possible approaches is the boring correct one, and where hiring for a skill you have not yet scoped is premature.
A generalist agency is the riskiest of the three, not because generalists cannot build this, but because the operational half is easy to underestimate if you have not run it before. Ask specifically how many AI integrations they have in production, still working, that they did not build in the last three months.
Whichever route, insist on the abstraction: the model behind your own interface rather than a provider's SDK called from application code. That keeps the decision reversible, which matters more than getting it right first time in a market moving this quickly.
What does a good first engagement look like?
One workflow, a measured baseline, a defined target, and a human in the loop. Small enough to ship in weeks and specific enough that everyone agrees afterwards whether it worked.
It should also leave you with assets rather than only a running system: the evaluation set, the prompts in version control, and documentation of what was tried and rejected. Those are what make the second integration faster, and they are what you lose if the engagement was scoped as a black box.
Judge the first project on whether it made the second one easier, not only on whether it worked.
Common questions
- What should you ask an AI integration partner before signing?
- Ask about failure paths rather than capability: what happens when the model is unsure, what it costs to run at ten times current volume, how you will know if accuracy drops in six months, what their evaluation set is, and which model with what licence. Every proposal describes the working case; few describe the rest, and that gap is where projects fail.
- Is a fixed price possible for AI integration work?
- For the build, often yes, provided the workflow is narrow and the data is understood. Running costs cannot be fixed the same way because inference scales with usage, so expect those quoted as a rate with a ceiling rather than a flat figure. A single price covering both usually means one of them has been guessed.
- Should we hire in-house or use a partner?
- Hire if AI is becoming a permanent part of your product, because drift, cost and model updates need an owner indefinitely and an agency build without one eventually becomes an orphan. Use a partner for the first one or two integrations, where the value is knowing which approach is the boring correct one and hiring for an unscoped skill is premature.
- How do you compare AI proposals fairly?
- Normalise whether running costs are included, whether evaluation and monitoring are in scope, and whether the price assumes clean data, since a data preparation pass is the most commonly excluded item and often exceeds the integration. Then ask each partner what they think the hard part is; the answers vary more than the prices and tell you more.
