Why more alerts make detection worse
Coverage is not safety, and treating a rule count as a security measure is the most expensive habit in this field. Detection capacity is a fixed number of human decisions per shift. A rule that fires forty times a week and is right once does not add security, it removes the attention that would have found something else. The question is never whether a detection is valid. It is whether you can afford to look at it.
Why does adding a detection make things worse?
Because precision collapses when the thing you are hunting is rare. A detection is a test, and the property that matters is not how accurate it is but what proportion of its alerts are real. Run a rule that is right ninety-nine times in a hundred against a million events a day that contain ten genuine ones, and you generate roughly ten thousand alerts of which ten matter. Nothing is wrong with the rule. The base rate did the damage.
This arithmetic is why vendor rule packs behave badly the week they arrive. The pack was written to be correct across every customer, which means it is tuned to the broadest reading of the technique it describes. Your environment supplies the event volume, and the volume sets the precision. Two organisations enabling an identical rule end up holding two different tools.
The practical consequence is that a detection's value is a property of your logs rather than of the detection. You cannot know whether a rule is worth having until it has run against thirty days of your own data and someone has counted the outcomes, which is the step almost every deployment skips in favour of enabling more.
What does one more alert actually cost?
The unit is analyst decisions, and a properly worked alert consumes a large slice of one: pull the context, check the account, check the host, check whether a change was scheduled, decide. Multiply by daily volume and you have derived your staffing requirement from your rule list, which is the wrong direction of travel. Most teams pick the rules and then discover the headcount.
The second cost is worse because it is invisible. An alert that is never worked is not free, it teaches everyone that the queue is decoration. Once a queue is understood to be optional, the failure mode is no longer a missed alert, it is a real alert that fired, was opened, and was closed as routine because ninety before it were routine. That is the pattern behind a striking number of published post-incident reports.
The third cost is orphaned logic. A rule whose author has left, whose alerts are always closed benign, and whose intent nobody can restate, cannot be safely tuned or safely deleted. It sits there consuming triage and defeating any attempt to reason about coverage, because nobody can say what it was supposed to see.
Which alerts deserve to interrupt a human?
The ones where a positive is almost certainly bad and the response is time-critical. Everything else belongs in a queue, in an enrichment pipeline, or nowhere. Sorting alerts by this question rather than by vendor severity is the single change that most reliably shrinks volume without shrinking coverage.
| Alert class | Precision in practice | Where it belongs | Failure when misplaced |
|---|---|---|---|
| Privileged identity change: role granted, MFA method added, admin sign-in from new country | High when scoped to privileged accounts only | Page | Scoped to all accounts, so the privileged one is lost in the volume |
| EDR behavioural detection, for example credential dumping or process injection | High | Page | Disabled wholesale after one noisy week, rather than by path |
| Signature match on a known-bad hash, domain or IP | High precision, poor coverage | Automated containment, then a ticket | Mistaken for a detection strategy in its own right |
| Threshold and anomaly: unusual data volume, rare parent process | Low alone, useful as context | Enrichment input, not an alert | Sent to a human who has no way to disprove it |
| Terminal shell opened inside a container | Low in any team that debugs in production | Queue, correlated with change records | Muted, taking genuine pod exec detection with it |
| Cloud control plane: audit logging disabled, instance role used from outside the network | Very high | Page | Buried under per-request noise from read operations |
How do you set a rule count you can staff?
Work backwards from the shift. Decide how many alerts a shift can genuinely work, subtract the volume your unavoidable high-precision detections already produce, and treat the remainder as the budget available for everything else. Then rank candidate rules by alerts per true positive and buy down the list until the budget is spent. Rules below the line are not deleted from existence, they are logged and searchable, just not alerted.
This reframes the tuning conversation from a technical argument into an allocation one, which is the only version of it that terminates. It also makes the trade explicit at the point of purchase: enabling a rule pack of several hundred detections is a staffing proposal, and it should be reviewed as one.
Ranking requires a number per rule, and that number is cheap to obtain and almost never collected. Alert count and disposition per rule over thirty days is a query against data you already hold. Without it every discussion about noise is a contest of anecdotes, usually won by whoever was on call most recently.
Why do containers and cloud break the arithmetic fastest?
Because they multiply. One image with an unexpected entrypoint running across three hundred pods produces three hundred alerts about one problem, and a horizontal scale event produces more while you are reading them. Any alerting on containerised workloads has to group by something stable, meaning the image digest or the workload identity, because the container ID is a fresh random string on every restart and makes deduplication impossible.
Kubernetes audit logging fails the same way from a different direction. Controllers and operators generate continuous get, list and watch traffic against the API server, so a detection written against API access without excluding those service accounts will bury everything else. The default rule sets shipped with runtime security tools have the same problem in estates where engineers legitimately exec into pods and where init containers write to paths the rules consider sensitive.
The container signals worth alerting on are the ones with naturally low volume. Process drift, where something is executing that was never in the image, is the strongest of them. Unexpected egress from a workload with a known destination profile is another, and exec into a production pod correlated against whether a change record exists is a third. Low natural frequency is not a limitation of these detections, it is the reason they survive contact with a real queue.
What test can you run this week?
Export every rule that fired in the last thirty days with its alert count and the disposition analysts recorded, then sort by alerts per confirmed true positive. The shape of that list is the honest state of your detection programme, and it usually shows a handful of rules producing the majority of the work and none of the findings.
For the worst offenders, disable rather than tune, for a stated period, with the decision written down and a date to revisit. Tuning a rule that has never produced a finding is optimising something with no demonstrated value. If a quarter passes and nothing was missed, you have your answer, and if something is missed you have a documented decision to revise rather than an argument to have.
Then look at the other end of the list: rules that fired zero times in thirty days. Either the behaviour genuinely never occurred or the rule is broken, and the two look identical from a dashboard. Run a controlled test of the technique with an open-source emulation tool and see whether the rule notices. Silence is not coverage, and this is the cheapest way to find out which kind of silence you have.
Common questions
- Why does a highly accurate detection rule still produce mostly false positives?
- Because of the base rate. Precision depends on how rare the target behaviour is, not on how accurate the rule is. A rule that is right ninety-nine times in a hundred, running against a million daily events that contain ten genuine ones, will produce thousands of alerts and only ten real findings. The rule is working as designed. The event volume in your environment is what determines whether the output is usable.
- How many detection rules should a team run?
- As many as the alerts they produce can be worked. Decide how many alerts a shift can genuinely investigate, subtract the volume your unavoidable high-precision detections already generate, then rank remaining candidate rules by alerts per true positive and enable down the list until the budget is used. Rules below the line stay searchable in logs without generating alerts, which keeps the data without spending the attention.
- Should you disable a noisy detection or tune it?
- Disable it if it has never produced a confirmed finding, because tuning something with no demonstrated value is misplaced effort. Record the decision, the reason and a date to revisit. Tuning is the right move for rules that do catch real activity but also fire on identifiable legitimate behaviour, where a narrow, owned and expiring exclusion removes the noise without removing the detection.
- Why do container alerts overwhelm a queue so quickly?
- Because they multiply by replica count. One bad image running across hundreds of pods generates hundreds of alerts about a single problem, and scaling events add more. Container IDs change on every restart, so alerts cannot be deduplicated unless the detection groups by image digest or workload identity. Kubernetes audit logs compound this, since controllers generate constant read traffic against the API server.
- What does it mean if a detection rule has never fired?
- It means one of two things that look identical on a dashboard: the behaviour has genuinely not occurred, or the rule is broken because a field mapping changed, a log source stopped arriving or the logic never matched anything. The only way to distinguish them is to safely reproduce the technique in your own environment and check whether the rule notices. Silence should never be reported as coverage.
