What should you actually monitor in containers?

What deserves an alert that wakes a human?

Only symptoms a user would notice, and only when a person can do something about them at that hour. In practice that is a small set: elevated error rate on a user-facing path, latency breaching an agreed threshold, a saturation signal with a hard ceiling such as disk space or a connection pool, and a workload that cannot start at all. Everything else belongs on a dashboard or in a report.

The test to apply to any existing alert is whether the last three times it fired, somebody changed something as a direct result. Alerts that have never caused an action are not warnings, they are noise with a claim to importance, and their real cost is that they train the on-call engineer to acknowledge without reading.

This is also why cause-based alerting ages badly. An alert on a specific known failure mode is useful for as long as that failure mode is the one occurring, and then it sits there forever describing a problem that has been fixed. Symptom-based alerting keeps working when the system changes underneath it.

Why do CPU and memory alerts mislead in containers?

Because the numbers mean something different once cgroup limits are involved. A container at ninety percent of its CPU limit is not in trouble; a container being throttled is, and those are separate measurements. Throttling is where the request latency actually comes from, and a utilisation alert will fire on the wrong workloads while staying silent on the throttled ones.

Memory is worse, because the figure most dashboards show includes page cache and does not correspond to what the kernel will act on. A container sitting at eighty percent of its memory limit may be entirely stable for months. The event that matters is the out of memory kill, which arrives without warning from the perspective of a percentage graph, and which shows up as a restart rather than as a gradual climb.

The rule that follows is to alert on the enforcement action rather than the level. Out of memory kills, sustained throttling, evictions and repeated restarts are events with a definite meaning. Percentages of a limit are for capacity planning, which is a conversation held on a Tuesday afternoon, not at three in the morning.

Which signals go where?

Three destinations, and putting a signal in the wrong one is the most common cause of an unusable alerting setup. The table is the shortest defensible version for a containerised service.

What drives the cost of container monitoring?

Cardinality and log volume, not the number of hosts. Every distinct combination of labels on a metric creates a separate time series, and container platforms attach identifiers that change on every deployment. Adding a pod name or a container identifier as a label multiplies your series count by the number of pods that have ever existed, which is why a bill can grow tenfold without any change in traffic.

This is the specific mistake worth checking for before you sign a contract with any platform. Labels should describe things that recur: service, environment, route, status class. Anything that is unique per instance or per request belongs in a trace or a log line, where it is stored once, rather than in a metric label, where it creates a series that is retained for months.

Logs then dominate whatever remains. The usual pattern is that debug-level output was enabled during an incident and never turned off, and the volume is invisible until the invoice arrives. Sampling successful requests while keeping all errors is the standard adjustment, and it typically costs nothing in diagnostic value because nobody reads the successful ones.

How much history do you actually need?

Two horizons, and conflating them is what makes retention expensive. Diagnosis needs high detail for a short window, because nobody investigates an incident with data from six months ago. Trend and capacity work needs a small number of aggregated series for a long window, because the question is whether something is growing.

So keep the detailed, high-cardinality data for a couple of weeks and downsample a deliberately small set of aggregates for a year or more. That set should be chosen rather than inherited: request rate, error rate, latency percentiles and a handful of business counters per service is usually enough to answer every question anybody actually asks a year later.

Traces are the exception where sampling beats retention. A representative sample of normal requests plus complete capture of errors and slow requests gives you the same diagnostic ability at a fraction of the volume, and the argument against it is almost always theoretical rather than a case somebody can name.

What audit can you run this afternoon?

Export every alert that fired in the last month and mark each one with what changed as a result. Most teams find that a small number of alerts account for the majority of pages and that a long tail has never produced an action. Delete the tail. This takes an afternoon, requires no new tooling, and improves incident response more than any platform migration.

Then run the test in the other direction, which is the more uncomfortable one. Take the last three incidents that customers noticed first, and check whether any alert existed that would have caught them. That gap is the real state of your monitoring, and it is invisible on any dashboard because dashboards only show what is already being collected.

When interviewing for this work, ask a candidate what they deleted from an alerting setup and what broke afterwards. Anyone who has genuinely run production has both a deletion they are proud of and one they regret, and the second answer is more informative than the first.