Model documentation and who actually reads it

Who actually reads model documentation?

Four readers, and each wants a different document. The person who has to decide whether to use the system at all, who needs limitations and failure modes and does not care about architecture. The person who operates it daily and has to know when to override it, who needs concrete instructions and examples of what wrong output looks like. The engineer maintaining or replacing it, who needs the training data, the dependencies and the version history. And the auditor or regulator, who needs evidence that the process happened when you say it did.

A single document written for all four serves none of them. The most common failure is a technical file that reads like a research paper, handed to an operations team who need to know what to do when the system flags a case incorrectly, and who consequently develop their own oral tradition about when to trust it. That oral tradition is your real documentation, and it is undocumented.

The useful discipline is to write the operator-facing document first. It is the shortest, it is the one that changes behaviour today, and drafting it forces you to answer the questions that the technical file usually evades: how often is this wrong, in which direction, and what should someone do about it.

What are the standard documentation artefacts?

Five, with different origins and different levels of formality. Model cards and datasheets for datasets come from the research literature and are voluntary conventions. System cards are a vendor practice describing a deployed system rather than a model. Technical documentation under Annex IV of the EU AI Act is a legal artefact for high-risk systems. Instructions for use are the provider's obligation to the deployer, and are the document a deploying organisation should be demanding rather than writing.

The overlap between them is substantial but not complete, so treating one as a substitute for another produces gaps that only appear under scrutiny. In particular, a model card describes a model and an Annex IV file describes a system, and the difference is everything that surrounds the model: the input pipeline, the thresholds, the human step, the fallback.

What does the EU AI Act require in the technical file?

For a high-risk system, Annex IV asks for a general description and intended purpose, the design and architecture, the data requirements including provenance and labelling of training and validation sets, the validation and testing procedures with the metrics used, the human oversight measures and the technical means by which output can be interpreted, the accuracy and robustness expected and the foreseeable unintended outcomes, the cybersecurity measures, the risk management system, a record of changes over the lifecycle, the standards applied, and the post-market monitoring plan.

The line item that catches providers is disaggregated performance and foreseeable unintended outcomes. The Act asks not only what the system's accuracy is but the circumstances in which it may produce risk, and for high-risk systems it asks about performance in relation to specific persons or groups where relevant. That cannot be written retrospectively from a single aggregate figure.

Separately, Article 13 obliges providers to give deployers instructions for use containing capabilities, limitations, accuracy metrics, the conditions that lead to risk, the oversight measures and the expected lifetime. If you are a deployer, that document is your entitlement. Asking for it during procurement, and reading what comes back, is a faster quality signal than any questionnaire.

What do you document when you did not train the model?

The system, not the model, and this is the position almost every organisation is in. You cannot document training data you have never seen, and pretending otherwise produces documents that are simply the vendor's marketing restated. What you can and must document is everything you control, which turns out to be most of what determines the outcome.

That list is concrete: which model and which version, with the date you pinned it and how you learn when it changes. What you send it, including the exact prompt or feature set, and where that input comes from. What you do with the output, including thresholds, filtering and any post-processing. Who or what checks it, and under what conditions. What happens when the vendor is unavailable. And what you tested before launch, on your own inputs, with the results recorded.

For the model itself, record what the supplier told you and attribute it as such. A line reading that the supplier states the model was evaluated on a particular benchmark, with the date and the document you were shown, is honest and defensible. A line asserting the same thing as your own finding is neither, and it is the first thing to unravel when the system is questioned.

How do you stop documentation going stale?

Attach it to the deployment rather than to the project, and make a change to production impossible without a change to the document. In practice that means the document lives in the same repository as the configuration, gets reviewed in the same change process, and names a version. Documentation that lives in a shared drive, separate from the thing it describes, is stale within a quarter without exception.

The second mechanism is a trigger list rather than a calendar. Annual review is the standard answer and it is weak, because models do not degrade annually. Define the events that force a review: the model version changes, the prompt or threshold changes, the input data source changes, the population served changes, a monitored metric crosses its bound, or an incident occurs. Any of those triggers a documentation update as part of the change itself.

The silent version change deserves particular attention when you use a hosted model. If your supplier can update the model behind a stable endpoint, your documented behaviour can become wrong without anyone at your organisation doing anything. Pin versions where the supplier allows it, and if they do not, record that as a known limitation rather than ignoring it.

What is the test for documentation that works?

Give the operator-facing document to someone who uses the system daily and did not help write it. Ask them one question: what should you do if the system tells you something you think is wrong. If they can answer from the document, it works. If they answer from experience, the document is decorative and the real procedure exists only in their head.

A second test, for the technical file. Ask whoever maintains the system to reproduce a result recorded in the documentation, using the artefacts referenced there. If the evaluation set cannot be located, or the model version is no longer identifiable, the documentation records a claim rather than evidence, which is a distinction an auditor will make even if nobody internal does.

Both tests take under an hour and neither requires an external reviewer. They are worth running before any assessment, because both failures are cheap to fix in advance and expensive to explain afterwards.