What to log, and what to stop logging
Start with the deletions. Most log estates are expensive because of a handful of high-volume sources that have never been used in an investigation, while the sources that answer real questions are missing or retained for weeks rather than months. Ingestion is priced by volume and investigations are constrained by coverage and retention, so the two goals pull in opposite directions and the resolution is selection rather than more budget.
What should you stop logging first?
The read path. Load balancer and web access logs at full fidelity, application debug output left enabled after a launch, object storage data events across buckets holding nothing sensitive, and Kubernetes API reads from controllers are, between them, the majority of the volume in a typical platform and are almost never the evidence that resolves a case. They can go to cheap object storage or be sampled, and neither choice weakens detection.
Duplication is the second target, and it is common enough to be worth an explicit check. Many estates ship the same events three times: once via an endpoint agent, once via the cloud provider's native pipeline, and once via a syslog collector installed years ago by someone who has left. Nothing detects better because a record arrived three times, but the invoice counts all three.
The honest way to find both is to ask which sources have been queried during an actual investigation in the last year. Query history in the platform will tell you. Sources nobody has ever searched are either unnecessary or unknown, and both conditions need fixing before the next renewal.
What is the minimum set that supports an investigation?
Six families, and the shape is consistent across estates: identity, process execution, egress, control plane, endpoint behaviour and email. Almost every intrusion narrative is assembled from those, and gaps in them are what turn a two-day investigation into a month of inference.
| Source | The question it answers | Keep quickly searchable | Common mistake |
|---|---|---|---|
| Identity provider sign-in and audit logs | Who authenticated, from where, with what method, and what changed afterwards | Yes, this is the highest value per gigabyte | Keeping sign-ins but not the audit trail of consent grants and role changes |
| Process execution with full command line | What ran on the host and what launched it | Yes | Enabling process auditing without the policy that records the command line |
| DNS and web proxy egress | Where a compromised host tried to talk to | Yes, though it is often the largest of the six | Logging at the resolver only, so the internal client is unidentifiable |
| Cloud and Kubernetes control plane audit | Which API calls changed the estate, by which identity | Yes for write and permission-changing operations | Logging reads at the same fidelity as writes, which drives most of the cost |
| EDR telemetry and detection events | What the behaviour looked like on the endpoint | Detections yes, raw telemetry usually vendor-side | Assuming the vendor's retention window matches your investigation needs |
| Mail and collaboration audit | Which messages and files an intruder read or forwarded | Yes, and check whether your licence tier includes access events | Discovering during an incident that mailbox access auditing was never enabled |
How long do you actually need to keep it?
Longer than most default retention, and the reason is that intrusions are frequently discovered well after they began. If your searchable window is thirty days and the initial access happened in month three, the investigation cannot establish scope, which is the question every regulator, insurer and customer will ask first. A window that does not cover plausible dwell time is a window that will fail at the only moment it is used.
The workable pattern is tiering rather than a single number. Keep the six families quickly searchable for a period measured in months, keep a cheaper archived copy for considerably longer, and accept that the archive is slower to query. Most platforms now sell this explicitly as lower-cost ingestion tiers with restricted query capability, or support querying data left in your own object storage.
Note that regulatory retention and investigative retention are different requirements and get confused constantly. A rule requiring you to retain certain records for years says nothing about how quickly they must be searchable, and satisfying it with expensive hot storage is a common and avoidable overspend. Where a sector rule prescribes retention, confirm the actual wording rather than the summary.
Why does ingestion cost grow faster than the estate?
Because most defaults are verbose and nothing removes them. Each new service arrives with its vendor's idea of useful logging, which is generally everything, and the cost lands in a central platform owned by a team that did not choose the setting. Growth is therefore multiplicative: more services, each chattier than the last, all billed by volume at the same rate as the six sources that matter.
The structural fix is to make ingestion a decision at onboarding rather than a default. When a new source is connected, someone should state which detection or investigation question it serves, what its expected daily volume is, and which tier it belongs in. That is a five-minute conversation that prevents a category of surprise no amount of later dashboard work recovers.
The second lever is normalisation before storage. Dropping fields that no query has ever referenced, and stripping repeated payload blobs from structured events, reduces volume without reducing answerability. This is unglamorous pipeline work and it is usually the difference between a platform that can afford twelve months of identity logs and one that cannot.
The third is to stop treating one destination as the only option. Detection needs a small, well-shaped set of events available quickly. Investigation needs a much larger set available eventually. Compliance needs a defined set available provably. Those are three different storage problems, and paying premium ingestion rates for all three is the single most common way a security budget is spent on something other than security.
What do container platforms log that you do not need?
Application stdout at debug level, which in a containerised estate goes straight into the same pipeline as security telemetry and often dwarfs it. Container platforms make this worse by collecting everything a workload prints by default, so a single service left in verbose mode can double a bill without anybody changing a security setting. Route application logs and security telemetry through separate paths with separate retention, because they are used by different people for different reasons.
For the Kubernetes API server, the audit policy file is the control that matters and it is worth writing deliberately. Metadata level is the default position for almost everything, including reads, and it must stay the level for Secrets, ConfigMaps, TokenReview and serviceaccounts/token. Raising those to Request or RequestResponse writes the secret material itself into the audit log, which is then shipped to a SIEM and retained by people who have no business holding it. Kubernetes says this in its own auditing documentation, and the upstream example policy logs configmap and secret changes at Metadata while logging pods at RequestResponse for exactly that reason. Reserve Request or RequestResponse for the non-sensitive writes where the body is the evidence: role and cluster role bindings, admission webhook configuration, and similar control plane changes. The default policies shipped by managed providers are usually more permissive than a detection team needs.
Exec and attach are a separate case, and turning their level up buys nothing. Requests to pods/exec and pods/attach are streaming protocol upgrades with no request body, so the only useful content, the container and the command arguments, arrives in the request URI, which Metadata level already records. More importantly, nothing typed inside the resulting shell session is captured by API server auditing at any level. If you need a record of what an intruder did once they were inside a container, that is runtime instrumentation, meaning eBPF-based tooling of the Falco class, not an audit policy setting.
Whatever you keep must carry stable identity. A container ID is meaningless an hour later, so every security-relevant record should include the image digest, the workload name and the namespace, or the investigation will be unable to answer which deployment the activity belonged to. Node-level process telemetry is generally a better buy than a sensor per container, since it survives the pod and costs less to run.
What audit tells you whether your logging is fit for purpose?
Pick a plausible scenario and try to answer it from logs alone, with a stopwatch running. A good one: a session token was stolen from an employee's browser and used from an unfamiliar network to read mail, download files from a shared drive, and create an API key in your cloud account. Nothing was malware, so the endpoint tooling has little to say.
Then answer five questions. When did the session first appear from a new location, and do you hold that record? Which mail items and files were accessed, or only which sessions existed? Which API key was created, by which identity, and what has it done since? Does your searchable window reach back far enough to prove when it started? And can you produce the answers without a vendor support ticket?
Teams commonly get through the first two and stop, which is precisely the finding worth having. Whatever you could not answer is the gap in your logging strategy, described in a form procurement understands, and it is far more persuasive than a coverage matrix because it came from a scenario nobody can dismiss as theoretical.
Common questions
- Which logs matter most for threat detection?
- Six families do most of the work: identity provider sign-in and audit logs, process execution with command lines, DNS or proxy egress records, cloud and Kubernetes control plane audit trails, endpoint detection telemetry, and mail and collaboration audit. Almost every intrusion narrative is reconstructed from these. Identity logs give the highest investigative value per gigabyte, because most modern intrusions involve credentials or sessions rather than malware.
- How long should security logs be retained?
- Long enough to cover plausible dwell time, which is usually longer than default retention. Intrusions are often discovered months after initial access, and if the searchable window does not reach back that far the investigation cannot establish scope. The workable pattern is tiered: months of quickly searchable data for the core sources, a longer and cheaper archive that is slower to query, and separate handling for anything a regulation requires you to keep.
- What logs can you safely stop collecting?
- Read-path telemetry at full fidelity is usually the largest safe reduction: web and load balancer access logs, application debug output, object storage data events for non-sensitive buckets, and Kubernetes API read operations from controllers. Duplicate feeds are the other target, where the same events arrive via an endpoint agent, a native cloud pipeline and an old syslog collector. Check query history to see which sources have ever been used.
- Why do SIEM costs rise faster than the size of the estate?
- Because verbose defaults accumulate. Every new service arrives with its vendor's logging turned up, the cost lands on a central platform owned by a team that did not choose the setting, and nothing removes it later. Growth is multiplicative rather than linear. The fix is to make ingestion an explicit decision at onboarding, stating which question the source answers, its expected volume, and which storage tier it belongs in.
- How should Kubernetes audit logging be configured?
- Through a deliberately written audit policy rather than a provider default. Keep Metadata level as the default, and keep Secrets, ConfigMaps, TokenReview and serviceaccounts/token there permanently, because Request or RequestResponse writes the secret values into a log you then ship to a SIEM and retain. Reserve Request or RequestResponse for non-sensitive writes where the body is the evidence, such as role binding and admission webhook configuration changes. Raising the level on pods/exec and pods/attach adds nothing, since those requests carry no body, the command arguments already appear in the URI at Metadata level, and nothing typed inside the session is audited by the API server at all. Every security-relevant event should also carry the image digest, workload name and namespace, because container IDs are meaningless after a restart.
- How do you test whether your logging is good enough?
- Run a scenario against your logs with a stopwatch. Take a stolen session token used from an unfamiliar network to read mail, download files and create a cloud API key, then try to establish when the session first appeared, exactly which items were accessed, what the new key has done, and whether your retention reaches the start. Whatever you cannot answer is your logging gap, expressed in terms procurement accepts.
