Mean time to detect is usually measured wrong
The number on the dashboard is almost never the time it took to detect an intrusion. It is calculated over incidents you did detect, starting from the moment an alert existed, which excludes both the intrusions you missed and the entire period before anything fired. Both distortions push the figure down. A programme can improve this metric by getting worse at finding old activity, which is the clearest sign that it is measuring something other than what it claims.
Why is the number you report misleading?
Because of survivorship. Mean time to detect can only be computed for incidents that were detected, so every intrusion still sitting unnoticed in your estate is absent from the denominator, and every one found by a third party months later either sits outside the process entirely or is excluded as an outlier. The metric describes your fast detections and is silent about your failures, which is the opposite of what a risk measure should do.
The distortion has a perverse direction. A team that starts hunting successfully and finds an intrusion that began four months ago will see its average detection time deteriorate sharply, and a team that stops looking will see it improve. Any metric that rewards not looking will eventually be optimised that way, usually without anyone intending it.
This does not make the number useless, it makes it a workload measure rather than a risk measure. Treated as a description of how quickly the process moves once something fires, it is informative. Treated as a statement about how exposed the organisation is, it is wrong in a direction that flatters everybody.
There is a second selection effect underneath the first. Incidents that produce a fast alert are usually the loud ones, meaning ransomware deployment, obvious malware and mass authentication failures, because loud activity is easy to detect. Quiet intrusions, which are the ones that matter most, are systematically under-represented in the average. So the metric is weighted towards the incidents you were always going to catch.
Where does the clock actually start?
There are four candidate start points and only two of them are automatically knowable. The intruder's first action is knowable only after forensic reconstruction, and often only approximately. The moment evidence first existed in your logs is knowable retrospectively, once you know what to search for. Alert creation and analyst acknowledgement are both recorded by tooling in real time, which is why almost every dashboard uses them.
So the figure most organisations call mean time to detect is the interval between an alert appearing and a human picking it up, which is a real and useful measure of queue health and is properly called time to acknowledge. Reporting it as detection time conflates the speed of your triage with the sensitivity of your detection, and those two improve through completely different investments.
The gap that matters is between evidence-available and alert-fired, because that interval is entirely within your control and is where detection engineering actually shows up. It cannot be dashboarded, since it can only be computed after an investigation has established when the evidence first existed. Computing it per incident, by hand, on a small number of incidents, tells you more than any automated average.
Which clocks are worth keeping?
Keep several, name them accurately, and stop presenting any single one as the state of detection. The table separates what each measurement can support from how it gets distorted.
| Measure | Clock starts | Clock stops | What it genuinely tells you | How it gets gamed |
|---|---|---|---|---|
| Time to acknowledge | Alert created | Analyst opens it | Whether the queue is staffed for its volume | Auto-acknowledgement, and bulk closing low severity items |
| Detection lag | Earliest evidence in your logs, established after investigation | First alert on that activity | Whether your detections cover the techniques used | Hard to game, and correspondingly hard to produce |
| Dwell time | Intruder's first confirmed action | Detection | Real exposure, distribution is what matters rather than the mean | Choosing a later start point when the early evidence expired |
| Time to contain | Incident declared | Attacker access removed | Whether authority and runbooks work out of hours | Declaring containment at isolation rather than at eviction |
| Internally found share | Not a duration | Not a duration | The most honest single indicator of detection capability | Counting a customer report as an internal finding |
| Retention headroom | Not a duration | Not a duration | Whether you could still measure any of the above in three months | Reporting archive retention as if it were searchable |
What should you report instead?
Start with the proportion of incidents your own organisation found, as against those reported by a customer, a partner, a law enforcement notification or the attacker themselves. It is a single number, it is difficult to distort, and it tracks detection capability more directly than any duration. A programme where most incidents arrive from outside has a detection problem regardless of how good its acknowledgement time looks.
Add detection coverage stated as evidence rather than as a percentage. A short matrix of techniques relevant to your architecture, marked as detected in testing, detected only with favourable conditions, or not detected, is defensible because each row was established by running the behaviour. Coverage percentages against a framework are not comparable between organisations and are usually inflated by counting rules rather than validated detections.
Then use distributions rather than means for anything that is a duration. Detection intervals are heavily skewed: most are minutes and a few are months, and a mean of those two populations describes neither. Report the median and the worst case, and the worst case is the number the board should be asked about.
Add one measure that most reporting omits entirely: whether you could still measure any of this in three months. Retention headroom, meaning the gap between how long your key sources stay searchable and how long intrusions plausibly sit undetected, determines whether the next incident can be scoped at all. It is a single comparison, it is knowable today, and it converts a storage line item into a risk statement executives can act on.
Why do containers and cloud make the clock unknowable?
Because the evidence expires faster than the intrusion is discovered. A compromised pod that was rescheduled has taken its filesystem with it, node-level telemetry may be gone with the node, and container logs are commonly retained for days. If the initial access happened before your retention window, the earliest evidence you can find is not the earliest evidence that existed, and every duration you calculate from it is an underestimate presented as a fact.
Short-lived infrastructure has the same effect in reverse for the response clock. Autoscaling can terminate the compromised instance before anybody investigates, which looks like containment and is actually destruction of evidence. Teams then measure a fast containment time for an incident whose scope they were never able to establish.
This makes retention a measurement decision rather than a storage decision, and it is worth stating that way to whoever owns the budget. If control plane audit and identity logs are held for a period shorter than plausible dwell time, then the organisation has chosen not to be able to measure its own exposure, and no amount of dashboard work recovers that.
What can you compute this week?
Take the last handful of real incidents, however few, and build a table by hand with five columns: who first noticed, what first noticed it, when the earliest evidence existed in your logs, when the first alert fired, and when access was actually removed. Do not average anything.
Read the rows. The distance between column three and column four is your true detection lag, and it is usually the largest number on the page. The distribution of column one tells you whether you are finding your own incidents. If any row cannot be completed because the data had expired, that is a retention finding, and it is more actionable than any metric target.
Repeat it after the next incident and keep the table permanently. Six rows of honest per-incident measurement will influence decisions in a way an averaged dashboard figure has never managed, precisely because each row is a story somebody in the room remembers, and nobody can argue that the mean is being dragged by an outlier.
Common questions
- Why is mean time to detect a misleading metric?
- Because it can only be calculated for incidents you detected. Intrusions still sitting unnoticed and those reported months later by a third party are excluded, so the figure describes your fast detections and stays silent about the failures. It also moves the wrong way: a team that starts finding older intrusions sees the average worsen, while a team that stops looking sees it improve.
- What is the difference between time to detect and time to acknowledge?
- Time to acknowledge runs from an alert being created to a human opening it, and it measures whether the queue is staffed for its volume. Time to detect should run from the moment evidence of the activity first existed to the moment an alert fired, which measures whether your detections cover the techniques used. Most dashboards report the first and label it the second, which conflates triage speed with detection sensitivity.
- What is the most honest detection metric to report?
- The proportion of incidents your own organisation found, as against those reported by a customer, a partner, law enforcement or the attacker. It is a single number, hard to distort and closely tied to actual capability. Pair it with a coverage matrix where each technique is marked as detected in testing, detected only under favourable conditions, or not detected, because those rows were established by running the behaviour rather than by counting rules.
- Should you use the mean or the median for detection times?
- The median, alongside the worst case. Detection intervals are heavily skewed, with most incidents caught in minutes and a small number found after months, so a mean of those two populations describes neither. Reporting the median and the longest case gives a board something it can act on, and the longest case is the figure that deserves the question rather than the average.
- How does short-lived infrastructure affect detection measurement?
- It destroys the evidence needed to measure. A rescheduled pod takes its filesystem with it, node telemetry can disappear with the node, and container logs are often kept for days. If initial access predates your retention window, the earliest evidence you can find is not the earliest that existed, so every duration you calculate is an underestimate. Autoscaling terminating a compromised instance also resembles containment while removing the ability to establish scope.
- How do you calculate real detection lag for an incident?
- After the investigation, search your logs for the earliest record of the intruder's activity, then compare that timestamp with the moment the first alert fired. That interval is detection lag, and it is entirely within your control. It cannot be automated because it depends on knowing what to search for, which is only clear afterwards, so compute it by hand per incident and keep the results as a permanent table.
