Cutting false positives without going blind
The dangerous tuning is the kind that works. Excluding a directory, a service account or a host stops the alerts immediately and provides no signal at all that you have removed a detection path an intruder can use. Reduction is easy; reduction you can still defend afterwards requires knowing which of three different problems you are solving, because they share a name and need opposite responses.
Which tuning moves quietly remove coverage?
Broad exclusions, and they are the ones the tooling makes easiest. Excluding an entire directory from endpoint inspection because a build agent writes there, excluding a service account from identity detections because it authenticates constantly, or excluding a host because it is noisy all achieve silence by creating a location where the behaviour is no longer observed. Intruders do not need to know why an exclusion exists to benefit from it.
Disabling a rule wholesale is the other common move, and it is more defensible than a broad exclusion despite feeling worse, because it is visible. A disabled rule appears in a configuration review. An exclusion buried in a policy applied to a device group does not, and in many products there is no report showing what has been excluded across the estate. If you cannot produce that list on demand, you do not know what you can see.
The pattern to be most suspicious of is an exclusion applied during an incident or a busy release week. Those are made under pressure by whoever had access, for a reason nobody wrote down, and they are permanent by default. Every mature detection programme eventually discovers a set of these and cannot reconstruct why any of them were added.
What is actually a false positive?
Three different things are given that label, and conflating them is why tuning discussions go in circles. First, the rule is wrong: it matches on a field your data populates differently, or its logic captures a pattern unrelated to the technique. That is a defect and the fix is to change the rule.
Second, the rule is right and the activity is authorised. A remote administration tool really did run, because your support team uses it. This is not a false positive at all, it is a detection working correctly against a behaviour you have chosen to permit, and the correct response is a narrow, documented exception tied to the specific circumstances that make it legitimate, not a change to the rule.
Third, the rule is right and nobody can act on it. Anomaly detections often land here: the observation may be accurate but there is no follow-up available that resolves it either way. These should not be alerts. They are context to attach to other alerts, and treating them as findings is how a queue fills with items that can only ever be closed as inconclusive.
The diagnosis takes a minute and is worth insisting on before any change is made. Ask whether the detected activity actually happened, and whether it was authorised. Yes and yes means an exception. Yes and no means an incident. No means the rule is defective. Teams that skip this question apply the same exclusion to all three situations, which fixes one, hides one and creates a permanent gap for the third.
Which technique fits which cause?
Match the response to the diagnosis. The table below is the decision most teams make implicitly and inconsistently, which is why two analysts facing the same noisy rule will produce different levels of blindness.
| Cause of the noise | Right technique | Wrong technique | Cost of getting it wrong |
|---|---|---|---|
| Rule logic matches unrelated activity | Fix the query, add a unit test with the benign case as a fixture | Exclude the hosts where it fires | The defect stays and spreads to every new host |
| Authorised tool used by a known team | Exception scoped to the tool signature, the account and the source, with an expiry date | Exclude the executable name globally | Anyone who renames a binary to match now operates unseen |
| Build or automation account triggers identity detections | Separate identity for automation, then detect on interactive use of that identity | Exclude the account from all identity rules | The most privileged non-human account in the estate becomes invisible |
| Container debugging by engineers | Correlate with change records, alert only on exec without one, keep the event either way | Disable shell-in-container detection | Loses the highest value detection available in a Kubernetes estate |
| Sidecar or agent writes to paths a rule considers sensitive | Exception keyed to the image digest, updated when the image changes | Exception keyed to the container or pod name | Breaks on every restart, so the exception is widened until it covers everything |
| Accurate anomaly with no possible follow-up | Demote to enrichment, surface it inside other alerts | Leave it alerting and close as inconclusive | Trains analysts that closing without an answer is normal |
How do you tune without losing the record?
Separate the alert from the event. In almost every platform you can stop something generating an alert while continuing to store it, and that distinction is the whole discipline. An investigation six months later can search data that was never alerted on; it cannot search data that was filtered at ingestion or suppressed at the sensor. Prefer alert suppression over collection suppression every time, even though collection suppression is what reduces the bill.
Then make exceptions accountable. Each one needs a stated reason, a named owner, the narrowest scope that solves the problem, and an expiry date that forces a cheap re-decision rather than a permanent hole. Expiry is the mechanism that does the real work here, because it converts a decision nobody will revisit into a five-minute confirmation on a schedule.
Keep the whole set in one place you can review, which for most organisations means version control rather than a console. Endpoint exclusion lists, identity rule scopes, runtime security exceptions and audit policy omissions are all forms of the same decision, and they are typically spread across four products with no combined view. Assembling that list once is often the most alarming afternoon a security team spends.
Restrict who can add one, too. Where any engineer with console access can create a permanent exclusion during a busy release, the list grows in ways nobody tracks and the security team learns about it during an investigation. Requiring the change to go through the same review as a rule change slows nothing down meaningfully and gives the decision an author.
How do you know whether you have gone blind?
By performing the technique and checking. Coverage claims based on configuration are inference; coverage established by safely running the behaviour and watching for the alert is evidence. Open-source emulation projects provide small reversible tests for a wide range of common techniques, and a weekly test against one important detection is enough to catch most silent failures within a month of them appearing.
Run those tests from somewhere your exclusions apply. This is the part usually got wrong: testing from a clean laboratory host proves the rule works in an environment that does not resemble production, while the exclusion that matters was applied to the build server group or the operations team's accounts. Test where the exceptions live, because that is where an intruder will end up.
After every real incident, check the exclusion list against the intruder's path. Ask directly whether any exception you hold would have covered any step they took. That question is uncomfortable, quick to answer and produces better tuning policy than any generic guidance, because it is grounded in behaviour that actually happened in your estate.
What weekly routine keeps this honest?
Three items, none of which takes long. Review the top few rules by alert volume and their disposition, so noise is addressed while the reason is still known rather than accumulating into a project. Confirm or remove every exception due to expire. Run one emulation test against one important detection and record the outcome.
Add one monthly item: reconcile the exception list across all products and compare it to last month. Growth is expected. Growth in exceptions with no owner, or scoped to whole hosts and accounts, is the signal that tuning has become erosion, and it is visible in that comparison long before it is visible in an incident.
The measure of success is not a lower alert count on its own. It is a lower alert count with a demonstrated ability to detect the same techniques you could detect before, which is a claim only continuous validation can support. Anything else is a quieter queue and an unknown amount of blindness.
Common questions
- What is the safest way to reduce false positive alerts?
- Stop the alert without stopping the collection. Most platforms let you suppress alerting while continuing to store the events, which preserves the record for a later investigation. Then scope any exception as narrowly as the problem allows, tying it to the specific signature, account and source rather than to a host, directory or whole service account, and give it an owner, a stated reason and an expiry date.
- Is an alert about authorised activity a false positive?
- No, and treating it as one leads to the wrong fix. If a remote administration tool ran because your support team uses it, the detection worked correctly against behaviour you have chosen to permit. The response is a documented exception covering the circumstances that make it legitimate, not a change to the rule logic. Rule logic should only change when the detection matches activity unrelated to the technique it targets.
- Why are broad exclusions dangerous?
- Because they create a location where the behaviour is no longer observed, and an intruder benefits from that without needing to know why it exists. Excluding a directory, a service account or a noisy host is effective at silencing alerts and invisible afterwards, since many products offer no estate-wide report of what has been excluded. If you cannot produce that combined list on demand, you cannot state what your detection covers.
- How do you handle container detections that fire on normal debugging?
- Correlate rather than disable. Keep the event in every case, and alert only when an exec into a production pod has no corresponding change record or approval. Where a sidecar or agent legitimately triggers a rule, key the exception to the image digest rather than the container or pod name, because names change on every restart and a name-based exception gets progressively widened until it covers everything.
- How can you prove that tuning has not created blind spots?
- By safely performing the technique and confirming an alert still appears, and by doing it from somewhere your exclusions actually apply. Testing from a clean laboratory host proves nothing about an exclusion applied to the build server group or the operations team's accounts. Open-source emulation projects provide small reversible tests for many common techniques, and one test a week catches most silent failures within a month.
- Why should detection exceptions expire?
- Because nobody revisits a permanent one. An expiry date converts a decision that would otherwise persist unexamined for years into a short confirmation on a schedule, at which point the owner either restates the reason or removes it. It also protects against the most common origin of exceptions, which is an urgent change made during an incident or a release week for a reason that was never written down.
