How to test an AI integration
Every test you have ever written assumes that the same input produces the same output. Models break that assumption, which is why teams that are otherwise rigorous ship AI features with no meaningful tests at all. The answer is not to give up on testing; it is to test different things.
Why don't normal tests work?
Because they assert equality, and equality is the wrong assertion. A model asked to summarise the same ticket twice may produce two different summaries, both correct. A test that compares against a stored string will fail on a perfectly good answer, so the team deletes the test, and the feature ends up with no coverage at all.
The second problem is that the thing you care about is not a value but a property. You do not need the summary to be a specific string; you need it to be under a length, to mention the customer's stated problem, and to contain no invented facts. Those are assertable. The exact wording is not.
So the shift is from asserting output to asserting properties of output, plus measuring quality across a set rather than per case.
What should you actually assert?
Structure first, and strictly. Does it parse? Does it validate against the schema? Are enum values from the allowed set? Are required fields present? These are deterministic, cheap and catch the majority of real production failures, because malformed output is far more common than subtly wrong output.
Then invariants. Length within bounds. No content in a field that should be a category. Numbers within a plausible range. A confidence score present. For extraction tasks, that every extracted value appears verbatim in the source, which is the single most effective automated hallucination check available.
Then behaviour on the paths that matter: what happens below the confidence threshold, on an empty input, on a very long input, on input in an unexpected language, and when the provider returns an error. Those are all deterministic tests of your code rather than of the model, and they are where most incidents actually originate.
| Layer | What it checks | Deterministic? |
|---|---|---|
| Schema validation | Parses, validates, enums legal | Yes |
| Invariants | Length, ranges, no field mixing | Yes |
| Groundedness | Extracted values appear in the source | Yes |
| Failure paths | Timeouts, errors, empty and huge inputs | Yes |
| Evaluation set | Accuracy across labelled cases | No, measured as a score |
| Regression on release | Score has not dropped | No, compared to baseline |
How do you build an evaluation set?
Take real inputs from the workflow, have a person who knows the domain label the correct output, and store them as fixtures. Fifty to a hundred cases is enough to be useful; a thousand is better but rarely necessary to start. The cases must be real, because synthetic inputs are systematically easier than production ones and will flatter the system.
Weight it towards the awkward cases deliberately. A set of clear, well-formed examples tells you almost nothing, since anything works on those. Include the ambiguous ones, the ones two colleagues would label differently, the ones with missing information. Those are where model changes show up first.
Then run the whole set on every prompt change, model change and provider change, and record the score. The absolute number matters far less than the direction: a score that drops after a prompt tweak is the signal the entire exercise exists to produce.
Why is accuracy the wrong target?
Because it treats every error as equivalent, and in a real workflow they are not. A support classifier that routes an urgent security issue to billing has failed far worse than one that routes a billing question to accounts, yet both cost one point of accuracy.
Measure the error you actually care about. For routing, that is usually the rate of high-severity misroutes. For extraction, invented values rather than missed ones. For anything with a human in the loop, the rate at which a person had to correct the output, which is the closest proxy to the real cost.
And measure the escalation rate alongside it. A system escalating almost nothing is not confident, it is overreaching, and its accuracy figure is hiding wrong answers that should have gone to a person.
What has to be monitored after launch?
Drift, because model behaviour changes without you doing anything. Providers update models behind the same version string, and your own data distribution shifts as the business changes. Something evaluated in March is not automatically correct in September, and nothing will tell you unless you are looking.
The practical approach is to sample continuously: take a small random share of live outputs each week, have someone review them against the same standard as the evaluation set, and track that score over time. Fifty a week is enough to see a trend.
Alongside it, watch three operational numbers that need no human review: schema validation failure rate, escalation rate, and latency at the ninety-fifth percentile. All three move before quality complaints arrive, which makes them the earliest warning available.
Common questions
- How do you test AI output when it is not deterministic?
- Assert properties rather than exact values. Check that the response parses and validates against a schema, that lengths and ranges are plausible, that enum values are legal, and for extraction that every extracted value appears verbatim in the source. Then measure accuracy as a score across a labelled evaluation set rather than pass or fail per case.
- How large does an AI evaluation set need to be?
- Fifty to a hundred real cases is enough to be useful, and more matters less than composition. Use genuine production inputs rather than synthetic ones, and weight the set towards ambiguous and awkward cases, because well-formed examples pass under almost any configuration and tell you nothing when something regresses.
- What is the best automated check for hallucination?
- For extraction tasks, verifying that every extracted value appears verbatim in the source document. It is fully deterministic, cheap to run on every response, and catches invented values immediately. For generative output there is no equivalent, which is why generative tasks need human sampling rather than only automated checks.
- Why is accuracy a poor measure for an AI integration?
- Because it treats all errors as equal when they are not. Routing an urgent security issue to billing is far worse than routing a billing question to accounts, yet both cost the same accuracy point. Measure the error class that carries real cost, plus the rate at which a human had to correct the output.
- Does AI output need monitoring after launch?
- Yes, because behaviour changes without you deploying anything: providers update models behind the same version string and your data distribution shifts over time. Sample a small share of live outputs weekly for human review, and watch schema failure rate, escalation rate and 95th-percentile latency, all of which move before complaints arrive.
