Incident response for AI and model failures
An AI failure will not page anyone. The service stays up, latency looks normal, error rates are flat, and the system produces confident output that happens to be wrong for a particular kind of case. By the time someone notices, the failure has been running for weeks and the pile of decisions already made is the actual problem. Existing incident processes are built around availability, which means they are structurally blind to the failure mode that matters most here.
Why is an AI incident different from an outage?
Three properties, and each breaks an assumption the standard process rests on. AI failures are usually silent: nothing throws an exception when a model classifies a category of documents wrongly, because from the software's point of view nothing failed. They are gradual, so there is often no moment of onset to correlate with a change. And they are selective, affecting one subpopulation or one input type while aggregate metrics stay within their usual range, which is precisely why aggregate monitoring misses them.
The consequence is that detection cannot rely on alerting from the system itself. It has to come from comparing behaviour against an expectation, whether that is a held-out reference set, the distribution of yesterday's outputs, or what a human reviewer would have said. All of these require deliberate instrumentation that nobody builds by accident.
The second consequence is more uncomfortable. Because the failure is silent and gradual, the incident is not really the failure, it is the accumulated set of decisions made while it ran. Restoring service resolves an outage completely. Restoring correct model behaviour resolves nothing for the people who were already refused, misrouted or wrongly flagged, and that backlog is where the legal and reputational exposure sits.
What counts as an AI incident?
Define it as any occasion where the system produced output outside the behaviour you documented, whether or not anyone was harmed. That definition is deliberately broader than a harm-based one because harm is often discovered late, and a process that only opens an incident once harm is proven starts the clock after the evidence has begun to age out of your logs.
Useful severity bands follow the affected population and the reversibility of the effect, not the technical dramatics. A model producing nonsense that a human catches every time is low severity. A model producing plausible errors that a human accepts is high severity even if the error rate is lower, because nothing stops it. A model that is right on average but systematically wrong for one group is high severity regardless of aggregate performance, because it is also a discrimination exposure and a different set of duties attaches.
Include categories that are not accuracy failures at all. Data leakage through outputs, where the system reveals information it should not have had or should not repeat. Prompt injection, where content in a document changes what an agent does. Availability of a supplier's model. And the one people forget: the system doing exactly what it was built to do, for a purpose nobody approved, because a team pointed it at a new use case.
How do you detect them?
By instrumenting for change rather than for error, and by accepting that the cheapest reliable detector is usually a small held-out set you re-run on a schedule. Most teams can implement two of these in a week and would catch the majority of real incidents earlier than they currently do.
| Signal | Catches | Typical lead time | Cost to run |
|---|---|---|---|
| Golden set re-run on a schedule | Model or prompt changes, supplier updates, regressions of known cases | Immediate on the next run | Low once the set exists; building it is the work |
| Input distribution monitoring | The world changing under the model: new formats, new customer mix, new language | Days to weeks before accuracy visibly drops | Low |
| Output distribution monitoring | Sudden shifts in decision mix, such as a category doubling overnight | Hours | Low |
| Human override rate | Reviewers quietly compensating for a degraded model | Often the earliest signal available | Low if reviews are already logged |
| Downstream outcome tracking | The failures nobody flagged, because the decision looked fine at the time | Weeks to months, but it is ground truth | Medium and worth it |
| Complaints, appeals and support tickets | Harms that only the affected person can see | Variable, and biased toward those able to complain | Low, but requires routing them somewhere |
What does containment look like?
Decide in advance, because the useful options are all configuration decisions someone has to have made possible. In descending order of preference: roll the model or prompt back to the last known good version; tighten the threshold so fewer cases are decided automatically and more go to a person; restrict the system to the subset of inputs where it is still known to work; fall back to the previous non-AI process; or disable the feature.
The first option is the one most teams assume they have and most do not. Rolling back requires that versions are pinned and retained, that prompts and configuration are versioned alongside code, and that the previous version is still available from the supplier. With a hosted model where the supplier changed the weights behind a stable endpoint, rollback may be impossible, and the only available containment is to reduce what the system is allowed to decide.
The fallback path deserves a rehearsal, once, before it is needed. Organisations retire the manual process when the model goes live, lose the people who ran it, and discover during an incident that falling back means a capability that no longer exists. Knowing that in advance changes the containment plan, and it is the kind of thing a thirty-minute tabletop exercise surfaces immediately.
What do you have to report, and how quickly?
Several clocks can start at once and they are not aligned, so establish which apply to each system before an incident rather than during one. Where personal data has been compromised, the GDPR obligation to notify a supervisory authority runs to 72 hours from becoming aware, with notification to affected individuals without undue delay where the risk to them is high.
Under the EU AI Act, providers of high-risk systems must report serious incidents to the relevant market surveillance authority. The Act defines a serious incident to include death or serious harm to health, serious and irreversible disruption of critical infrastructure, breach of Union law obligations intended to protect fundamental rights, and serious harm to property or the environment. Reporting is required immediately after a causal link is established and in any event within fifteen days of becoming aware, shortened for widespread infringement or critical infrastructure disruption, and with a separate window where a death is involved. Confirm the current wording before relying on any of these figures.
Then the sector clocks: financial services, medical devices, critical infrastructure and employment regulators each have their own duties, several of which predate AI regulation entirely and apply regardless of whether a model was involved. And your contractual clocks, which are frequently the tightest, since enterprise customers commonly require notification within a day. The practical output of this section is a single page per system listing who must be told, by when, and who signs it off.
What about the decisions already made?
This is the part that gets deferred, and it is the part that determines how the incident is judged afterwards. Start by bounding the window: when did the behaviour change, and what is the earliest point you can prove it was correct. Absent good logging, the honest answer is often that you cannot tell, which is itself a finding and an argument for retaining inputs, outputs and model versions long enough to reconstruct a decision.
Then identify the affected population and decide, explicitly, on a remedy for each of three groups: those whose decision would have been different and was worse for them, those whose decision would have been different and was better for them, and those where you cannot determine either way. The middle group is the one that generates the most argument internally, and reversing a favourable decision made in error is a legal question rather than an engineering one.
Reprocessing is usually the right answer for the first group and it is rarely as simple as re-running the model, because circumstances have moved on: applications were withdrawn, customers went elsewhere, deadlines passed. Assume the remedy involves contacting people and offering a genuine reconsideration, budget for that as the main cost of the incident, and record the decision not to reprocess with equal care where that is the conclusion, because it is the one that will be examined.
Common questions
- What is an AI incident?
- Any occasion where an AI system produced output outside its documented behaviour, whether or not harm has been established. Defining it by output rather than by proven harm matters, because harm is usually discovered late and a process that waits for proof starts the investigation after logs have begun to expire. The definition should also cover data leakage through outputs, prompt injection, supplier unavailability and use of the system for an unapproved purpose.
- Why don't normal monitoring tools detect AI failures?
- Because model failures are silent, gradual and selective. Nothing raises an exception when a model classifies a category of inputs wrongly, so availability and error-rate monitoring stay green. Degradation often has no moment of onset to correlate with a deployment. And a failure affecting one subpopulation can leave aggregate accuracy inside its normal range, which is exactly why aggregate monitoring misses the cases that carry legal exposure.
- How do you detect model degradation in production?
- Compare behaviour against an expectation rather than waiting for errors. The most reliable cheap detector is a held-out reference set of cases with known correct answers, re-run on a schedule. Add monitoring of input and output distributions to catch the world changing under the model, watch the rate at which human reviewers override the system, and route complaints and appeals somewhere they are counted rather than only resolved.
- How quickly must an AI incident be reported?
- Several clocks can run at once. Where personal data is compromised, GDPR requires notification to a supervisory authority within 72 hours of becoming aware. Under the EU AI Act, providers of high-risk systems must report serious incidents to market surveillance authorities immediately after establishing a causal link and within a defined maximum period, shortened where critical infrastructure or a death is involved. Sector regulators and enterprise contracts frequently impose tighter windows still.
- Can you roll back an AI model?
- Only if you prepared for it. Rollback requires that model versions are pinned and retained, that prompts and configuration are versioned alongside code, and that the previous version is still available from the supplier. Where a hosted supplier has changed the model behind a stable endpoint, rollback may be impossible and the available containment is to reduce what the system is permitted to decide, route more cases to people, or fall back to the previous process.
- What do you do about wrong decisions an AI system already made?
- Bound the window by establishing when behaviour changed and the last point it was provably correct, which requires having retained inputs, outputs and model versions. Then split the affected population into those harmed by the error, those who benefited from it and those where the outcome cannot be determined, and decide a remedy for each explicitly. Reprocessing usually means contacting people and offering genuine reconsideration rather than simply re-running the model.
