Measuring a DevSecOps programme without counting scans

Which metrics look useful and are not?

Four appear in almost every early dashboard and none of them survives scrutiny. Number of scans run measures how often a pipeline executed, which is a function of commit volume. Findings closed rewards closing easy findings and moves sharply when a base image is rebuilt, with no relation to effort or risk. Total open findings moves when vulnerability databases are updated rather than when your engineering changes. Training completion percentage measures attendance.

Coverage percentage is the subtle one, because it sounds like exactly the right thing. The problem is what it counts: repositories with a scanner configured is not the same as repositories where the scanner runs on every change, which is not the same as repositories where someone reads the result. A programme can report ninety per cent coverage while the findings from all of it go to an unmonitored channel.

The test for any proposed metric is to ask what someone would do to improve the number without improving security, and whether that action is easier than the real work. If it is, you have created an incentive rather than a measurement. Every metric below fails this test somewhere, which is why the section states where.

What should you measure instead?

Six measures, each describing a capability rather than an activity. Together they answer the questions an executive actually has: could we respond quickly, do we know what we run, and is the process being followed or worked around.

Why is time to deploy a one-line patch the best single number?

Because it is the constraint on every security response you will ever need to make. When a widely used library gets a critical advisory on a Friday afternoon, nothing about your scanner coverage matters, and the only question is how long it takes to get a version bump through your pipeline into production. A team that can do that in an hour is in a different risk position from one that needs a release window, whatever their finding counts say.

It also has the useful property of being a delivery metric rather than a security metric, which means improving it is aligned with what the engineering organisation already wants. Change lead time is familiar from ordinary delivery measurement, and reading it as a security capability lets you argue for pipeline investment without framing it as a security tax.

Measure it as a drill on an unrehearsed service, not as an average from historical data. Averages are dominated by the services that release constantly and hide the one that has not deployed since last year, which is invariably the one that will hold up your response. Pick a service at random each quarter and time the whole thing including approvals.

How do you measure whether controls are actually enforced?

By reading the enforcement point, never the tool that produces the artefact. The number of images signed tells you your build works. The number of deployments where a signature was verified and could have been refused tells you whether the control exists. Those two figures diverge more often than anyone expects, usually because verification was enabled in one cluster and not the others.

Do the same for policy. Count policy evaluations that returned a denial and what happened next, rather than the number of policies defined. A policy set with hundreds of rules and no denials in a quarter is either perfectly complied with, which is implausible, or not running where you think it is. This is a check you can perform in an afternoon and it regularly finds a control that lapsed during an unrelated migration.

The principle is to measure at the point where something is refused, because everything upstream of a refusal is potential and potential accumulates without changing outcomes. It is also why an escape rate is worth tracking: significant findings discovered after deployment, by a test, a customer or an external report, is the only signal about what the pipeline misses that is not self-reported.

What do you report upward, and how often?

Three or four numbers, monthly, with the same definitions every time. Time to deploy a patch, share of production artefacts within your freshness target, open exceptions with the oldest one named, and the count of unowned production components. Each is a single figure, each has an obvious direction, and none of them is a total that moves when a vulnerability database updates.

Add one narrative item, which is the incident or near miss of the month and what changed as a result. Boards remember specifics and forget trends, and a programme that reports only improving numbers with no stories is eventually and correctly suspected of managing the numbers. Naming a case where a control failed and describing the fix buys more credibility than any dashboard.

Resist adding metrics over time. Every reporting pack drifts towards a page of numbers nobody reads, and the drift usually happens because someone asked a one-off question and it became permanent. Answer the question separately and keep the standing set small enough that the audience can hold it in their head between meetings.

What can you calculate this week?

Two numbers, both available from data you already hold. First, the age distribution of container images currently running in production, taken from build timestamps. If the oldest is measured in months, you have found your most important remediation item, and it is a pipeline task rather than a triage one. Second, the number of production components you can route to a current owning team from stored metadata alone.

Then run the patch drill. Pick a service nobody has deployed recently, make a trivial dependency bump, and time it from commit to running in production including every approval. Record where the time went, because the answer is almost never the pipeline itself and almost always a wait for a person or a window.

Those results will tell you more about the programme than a quarter of scanner reports, and they are hard to argue with. Numbers a team can dispute get disputed; numbers taken from build metadata and a stopwatch tend to produce a decision.