Why observability is a build-time decision
Observability is sold as something you install and it is mostly something you decide in application code, usually about a year before you need it. Agents, dashboards and log pipelines can all be added to a running system in an afternoon. What cannot be added afterwards is the context inside each record: which customer, which request, which version, which upstream call. Without that, you own a large volume of data and still cannot answer a question about one user.
Can you add observability later?
You can add collection later. You cannot add context later, and context is the whole of the value. Installing an agent gives you request counts, latency percentiles and error rates within an hour, and those are genuinely useful for knowing that something is wrong. They will not tell you who it is wrong for, which is the first question anyone asks.
The difference is that context has to be attached at the moment the event happens, by code that knows what it is doing. A log line written without a request identifier cannot be joined to anything afterwards. A trace that stops at a queue boundary cannot be extended retroactively. A metric recorded without a version label cannot answer whether the new release caused the change, no matter how good the query language is.
This is why the decision belongs in the build rather than in procurement. The vendor is replaceable and the instrumentation is not, which is also the argument for emitting through a vendor-neutral interface so that changing supplier is a configuration change rather than a re-instrumentation project.
What does each signal actually answer?
Three signals with three distinct jobs, and the common failure is asking one of them to do another's work. Metrics tell you the shape of the system over time and cost almost nothing to keep. Traces tell you where the time went in one request and are expensive at full volume. Logs tell you what happened in detail and cost in proportion to how much you write.
Reading the fourth column below is the useful exercise, because most observability spending is caused by trying to get an answer from a signal that structurally cannot provide it.
| Signal | Answers | Cost driven by | Cannot tell you |
|---|---|---|---|
| Metrics | Is it broken, how badly, and since when | Number of distinct label combinations | Which specific request or customer was affected |
| Traces | Where the time went across services in one request | Sampling rate multiplied by span count | Anything about requests you did not sample |
| Structured logs | What the code decided and why, in detail | Volume ingested and how long it is retained | Aggregate behaviour, without expensive queries |
| Continuous profiles | Which code paths consume CPU and memory | Sampling frequency, usually modest | Anything about correctness or business outcomes |
What has to be in the code?
Four things, and they take a day to add at the start of a project and a quarter to add later. A request identifier generated at the edge and carried through every log line and outbound call. Trace context propagated across every boundary, including message queues and background workers, which is where most instrumentation quietly stops. A consistent set of attributes on every record: service name, version, environment, and the tenant or customer identifier. And structured output, so fields are queryable rather than parsed out of a sentence with a regular expression.
The queue boundary deserves particular attention because it is where traces almost always break. A request that publishes a message and returns looks complete, and the work triggered by that message appears as an unrelated orphan trace some seconds later. Joining them means putting the trace context into the message headers and restoring it in the consumer, which nothing does for you unless the library has been configured to.
The customer identifier is the one that returns the most value per line of code. Every incident begins with a specific complaint, and the ability to filter to one account across every service turns the first thirty minutes of an investigation into a single query.
Why does cardinality decide the bill?
Because a metric with labels creates a separate stored series for every combination of label values, and the storage and query cost grows with the number of series rather than with the number of requests. Adding a customer identifier as a metric label on a system with a hundred thousand customers creates a hundred thousand series per metric, and doing that to a handful of metrics is how an observability bill grows faster than the traffic it is measuring.
The correct division is that metrics carry low-cardinality dimensions only, meaning things with a small and stable set of values: endpoint, status class, region, version. High-cardinality identifiers belong on traces and logs, which are stored as individual events and are designed to be filtered by anything.
The practical control is a review step on new metric labels and a check for unbounded values, which are usually user input reaching a label by accident: a URL path containing an identifier, an error message used as a label, a customer-supplied header. Any of those will generate series without limit, and the bill arrives before the alert does.
What should you be able to answer?
Five questions, and if your setup answers them you have enough regardless of which tools you bought. Is the system healthy right now. When did it stop being healthy. Which component is responsible. Which customers are affected. And what changed immediately before it started.
The last one is the most frequently missing and the cheapest to fix. Emitting the deployed version as an attribute on every metric, log and trace lets you overlay releases on any graph, and a very large share of incidents are answered by that overlay alone. Without it, every investigation starts by asking three people what was deployed and when.
There is a sharper version of the test: take a real complaint with a timestamp and an account, and try to find that customer's failing request. If it takes more than five minutes, the gap is instrumentation rather than tooling, and no dashboard will close it.
Where do teams overspend?
Four places. Retaining everything at full fidelity for months, when incident investigation almost always happens within days and anything older is being kept for compliance reasons nobody has checked. Debug-level logging left on in production after an investigation, which multiplies volume and is rarely noticed because it degrades the bill rather than the service. Running two overlapping agents because a new platform arrived and the old one was never removed. And logging a line per request on a high-traffic path, where a metric would have answered the same question for a fraction of the cost.
The structural fix is tiered retention rather than blanket reduction: keep everything briefly, keep a sampled subset longer, and keep aggregates indefinitely because they are small. That preserves the ability to investigate recent problems and the ability to see year-on-year trends, and drops only the expensive middle where full-fidelity old data sits unread.
Sample traces intelligently rather than uniformly. Head-based sampling at a fixed percentage discards errors at the same rate as successes, which is the wrong way round. Keeping every failed and slow trace and a small sample of the rest costs less and is more useful, because nobody investigates a fast successful request.
Common questions
- Can you add observability to an existing application?
- You can add collection quickly, and an agent will give you request rates, latency and error counts within an hour. What cannot be retrofitted is context: a log line written without a request identifier cannot be joined to anything later, and a trace that stopped at a queue boundary cannot be extended retroactively. Collection is a purchase; context is a code change made earlier.
- What is the difference between metrics, logs and traces?
- Metrics describe the shape of the system over time and are cheap to keep, but cannot identify an individual request. Traces show where time went across services within one request, and only cover what was sampled. Logs record what the code decided in detail and cost in proportion to volume and retention. Each answers a different question, and most overspending comes from asking one to do another's job.
- Why is metric cardinality expensive?
- Because each unique combination of label values creates its own stored series, so cost grows with distinct combinations rather than with request volume. Adding a customer identifier as a label on a system with a hundred thousand customers creates a hundred thousand series per metric. High-cardinality identifiers belong on logs and traces, which are stored as individual events and designed to be filtered by anything.
- What should every log line contain?
- A request identifier generated at the edge, the trace context, the service name, the deployed version, the environment, and the tenant or customer identifier where one applies, all as structured fields rather than prose. The version attribute in particular resolves a large share of incidents on its own, because it allows releases to be overlaid on any graph without asking anyone what was deployed.
- How should traces be sampled?
- Prefer keeping every failed and unusually slow trace plus a small sample of ordinary ones, rather than a uniform percentage of everything. Fixed head-based sampling discards errors at the same rate as successes, which removes exactly the traces anyone would want to look at. Nobody investigates a fast successful request, so paying to store a representative sample of them is the wrong end of the budget.
