Detection and response as the real deliverable
The vulnerability list in a red team report is the part you should read last. Individual flaws get patched and return; the reason nobody noticed a fortnight of activity does not change on its own. Treating detection and response as the deliverable changes what you buy, what the report must contain, and what has to be switched on beforehand, and the last of those three is where most engagements quietly lose their value.
Why is the vulnerability list the least valuable part?
Because it is the part that regenerates. The specific unpatched service, the over-permissioned role, the credential in a repository: each gets fixed within weeks, and something similar appears next quarter through ordinary change. The capability to notice when it is being abused persists across all of them, which makes it the higher-order investment even though it is harder to point at.
A red team is also a poor instrument for finding vulnerabilities. It stops looking once it has a path, deliberately avoids noisy enumeration, and therefore returns a small and unrepresentative sample. Scanning and scoped penetration testing find more, faster and cheaper. Paying red team rates for a defect list means buying the worst available version of that product.
The useful test is to ask what the report would tell you if every named vulnerability had already been fixed. If the answer is nothing, the engagement measured your patching. If the answer is a set of statements about which activity was invisible, which was recorded but unalerted, and how long containment took, the engagement measured your organisation.
What has to be true before detection can be the deliverable?
Four preconditions, all of which are your responsibility rather than the supplier's. Logs from identity, endpoints, cloud control planes and the container platform must exist, be retained for longer than the engagement window, and be searchable by someone after the fact. The red team must be contractually required to record a timestamp for every action. There must be a defender-side record of alerts and cases with timestamps. And someone must own the reconciliation afterwards.
Retention is the precondition that fails most often. An engagement running over eight weeks against a platform holding seven days of logs cannot be reconciled, because by the time anyone looks the evidence for the early phases has aged out. Check the actual retention per source rather than the policy, since sampling and quota limits often reduce it silently.
The cheap fix is to snapshot rather than extend. Before the engagement starts, arrange to export the relevant log sources to durable storage for the duration, then reconcile against the export. That converts a platform licensing problem into a storage cost, and it is the single most useful preparatory step available.
Which telemetry has to exist, and why is it usually missing?
The layers below are the ones an intrusion crosses. Endpoint and identity are commonly well covered. Everything from the cloud control plane down is commonly not, which is why the middle of most red team timelines is blank.
| Layer | Telemetry needed | Why it is usually missing | Cheap first step |
|---|---|---|---|
| Identity and authentication | Sign-ins, factor changes, consent grants, role assignments | Usually present, but retained briefly | Extend retention for privileged accounts |
| Endpoint | Process, command line, credential access events | Present, though coverage of servers lags laptops | Audit which servers have no agent at all |
| Cloud control plane | API audit trail across every account and region | Enabled in one account, not organisation wide | Turn on organisation-wide trails to central storage |
| Kubernetes control plane | Audit events for exec, create, secret reads, RBAC changes | Audit policy off, sampled, or not shipped anywhere | Enable audit logging and ship it to your log platform |
| Container runtime | Process execution and file activity inside containers | No node sensor, because agents do not run in pods | Deploy a node-level runtime sensor to one cluster |
| Network egress | Outbound flows attributed to a workload identity | Flows logged per node, so pods are indistinguishable | Add workload attribution, or egress through a proxy |
| CI/CD and source control | Pipeline runs, secret access, deployment approvals | Treated as engineering tooling, not a monitored system | Forward pipeline and repository audit logs |
Why do containers break the detection story?
Because the assumptions behind endpoint monitoring do not hold. An agent expects a long-lived host with a stable name, a filesystem it can watch and a process tree it owns. A container is short-lived, shares the host kernel, is named after a pod that will not exist tomorrow, and does not run your agent inside it. Detection therefore has to come from the node and from the control plane, which is a different tooling decision that many organisations have not made.
The consequences show up in three specific ways. Process execution inside a container is invisible without a node-level sensor, typically kernel-instrumented, so the entire execution phase of an intrusion leaves no endpoint record. Control plane actions such as exec into a pod, creating a privileged workload or reading a secret are only recorded if audit logging is configured and shipped, and in several managed distributions the useful verbosity is not the default. And attribution decays: an alert naming a pod that was replaced six hours ago is difficult to investigate unless you retained the workload metadata alongside the event.
There is also a base rate problem that no tool solves for you. Container platforms are extremely noisy by nature, with images pulled, containers restarted and jobs scheduled constantly, so a rule that fires on unusual process execution will drown you unless it is scoped to a workload baseline. That is why container detection work is mostly labelling: knowing which images should exist, which service accounts should call which APIs, and which egress destinations are legitimate. Teams that maintain that inventory get useful alerts, and teams that skip it end up muting the sensor within a month.
What do you do with a detection gap?
Sort it into two piles, because they cost different amounts. A gap where the event was recorded and nothing fired is a rule, and a rule is an afternoon plus a test. A gap where nothing was recorded at all is a pipeline change: an agent to deploy, an audit policy to enable, a log source to route, a storage bill to approve. Reporting these two together as findings guarantees that the cheap ones are delayed by the expensive ones.
Then own the rules internally, in version control, with a test that proves each one fires against a documented execution of the technique. Detections written by a supplier and left in a console rot within a release cycle, and nobody notices because the absence of an alert looks identical to the absence of an attack. The test is what distinguishes them.
Sequence the two piles deliberately. Ship the rules first, because they are quick and they demonstrate movement, but start the pipeline work in the same week, since it carries procurement, storage and platform dependencies that will otherwise push it past the quarter. The common failure is a programme that closes forty rule-shaped items and never routes the missing log source.
Watch alert quality as you go. Every new rule competes for the same triage attention, and a detection programme that doubles alert volume has made response slower while looking more capable. If a rule cannot be scoped to a rate a human can absorb, it belongs in a hunting query or a weekly report rather than in the alert queue, and saying so out loud is a sign of a mature detection function.
How do you know your response works, not just your detection?
By rehearsing the actions, with timings, on real systems. Detection ends at an alert; response is a set of specific things somebody has to do under pressure, and each of them either works today or does not. Isolate a laptop from the network. Revoke a user's sessions and force reauthentication. Rotate a cloud access key that a running service depends on. Revoke a Kubernetes service account token and confirm the workload's existing connections drop. Evict a pod, block an image from being redeployed, and rebuild it from a trusted source.
Time each one and record who is authorised to do it out of hours. The common failure is not technical: the person who can approve isolating a production service is asleep, and the on-call engineer will not do it without approval. That gap is why detected intrusions still progress, and it is fixed with a written standing authority rather than with tooling.
The container-specific rehearsal worth running this quarter is token revocation. Most teams have never tested it and are surprised by the result, because revoking a service account credential does not necessarily terminate sessions already established with it, and rebuilding the workload may quietly restore the same broad permissions. Run it once in a controlled window, write down what actually happened, and you will have found more real risk than the average vulnerability report contains.
Common questions
- What is the main deliverable of a red team engagement?
- The detection and response assessment, not the vulnerability list. Individual flaws get patched and reappear through ordinary change, while the reason nobody noticed a fortnight of activity persists. A useful test: if every named vulnerability had already been fixed, would the report still tell you something? It should still say what was invisible, what was recorded but unalerted, and how long containment took.
- What logs do you need before a red team engagement?
- Identity, endpoint, cloud control plane, Kubernetes audit, container runtime, network egress with workload attribution, and CI/CD activity, all retained beyond the engagement window and searchable afterwards. Check actual retention per source rather than the policy, since sampling and quotas often reduce it silently. Exporting the relevant sources to durable storage for the duration is cheaper than extending platform retention.
- Why is detection harder in containers than on servers?
- Endpoint agents assume a long-lived host with a stable name and a process tree they own. Containers are short-lived, share the host kernel, and do not run your agent inside them, so execution telemetry must come from a node-level sensor and control plane actions must come from audit logging that is often not verbose by default. Alerts also lose attribution once the pod they name has been replaced.
- How do you prioritise detection gaps after a red team?
- Split them by cost. A gap where the event was recorded but nothing fired is a rule plus a test, roughly an afternoon. A gap where nothing was recorded at all is a pipeline change: an agent, an audit policy, a log route and a storage bill. Reporting both as one list of findings guarantees the cheap fixes wait behind the expensive ones.
- How do you test incident response rather than detection?
- Rehearse the specific containment actions and time them. Isolate a laptop, revoke user sessions, rotate a cloud key a live service depends on, revoke a Kubernetes service account token, evict a pod and block its image from redeployment. Record who is authorised to approve each action outside working hours, because the usual failure is authority rather than capability.
- Does revoking a Kubernetes service account token stop an attacker?
- Not always, and most teams have never tested it. Revoking or rotating the credential does not necessarily terminate sessions already established with it, and redeploying the workload can restore the same broad permissions immediately. Run the rehearsal once in a controlled window and record what actually happened, because the answer is specific to your cluster version and configuration.
