Why observability comes before the split, not after
In a monolith, a failure produces a stack trace that names the line. Split the same system and that stack trace becomes a timeout in one service caused by something in another, with no link between the two unless somebody built one. Teams routinely plan the split and schedule the instrumentation afterwards, which means the riskiest months of the migration are the months you cannot see.
What breaks first without observability?
Diagnosis. The time to understand an incident grows with the number of services, not with the size of the problem, because each additional service is another place the cause might be and another set of logs somebody has to open. A one-line bug can take an afternoon to locate when four teams each have to prove it was not them.
The pattern that follows is predictable: incidents become meetings. Without a shared trace, the only way to work out what happened is to gather the people who own each service and compare timestamps in different formats and time zones. That is not an engineering process and it does not get faster with practice.
The specific missing capability is almost always the same. Nobody can take one failed customer request and show its whole path. Every other observability gap is survivable for a while; that one makes the system opaque immediately.
What is the minimum before the first split?
Four things, and the first is non-negotiable. A trace context generated at the edge and propagated through every hop, including asynchronous ones. Most teams get this right for HTTP and forget the broker, so the trace ends at the publish and a new one starts at the consumer, which is precisely where the interesting failures live. Message headers carry the context; it just has to be written and read deliberately.
Second, structured logs that include the trace id, so a log line can be pivoted to a trace and back. Free text logs do not survive a distributed system, because correlating them means writing regular expressions against four different formats during an incident.
Third, request rate, error rate and duration per endpoint for every service, with duration recorded as a distribution rather than an average. Averages hide the failure mode that matters: a small proportion of very slow requests is what exhausts connection pools, and it does not move the mean. Fourth, health checks that distinguish 'starting' from 'ready' from 'alive', because an orchestrator that cannot tell the difference will send traffic to a process that is not ready or restart one that is merely busy.
What does each signal actually answer?
The three signals are not alternatives and buying only one is the usual mistake. Each answers a different question and is priced differently.
| Signal | Answers | Weak at | Cost driver |
|---|---|---|---|
| Metrics | Is something wrong, and since when | Explaining why, or naming an affected customer | Label cardinality, not request volume |
| Traces | Where the time went in this request, across services | Rare events, if sampling discards them | Sampling rate and retention |
| Logs | What exactly happened in one service at one moment | Answering anything that spans services alone | Volume, and it grows with traffic |
| Health checks | Should this instance receive traffic right now | Anything about correctness | Negligible, if they do not call dependencies |
Why is client-side measurement the one everyone skips?
Because the server's own timing looks fine. A service reporting a twenty millisecond response time can be seen by its caller as taking most of a second, and the difference is real: connection pool wait, DNS, TLS handshake, queueing before the request is picked up, and retries that the server records as separate healthy requests.
So every outbound dependency needs to be measured from the caller: latency distribution, error rate, timeout count and retry count, tagged by which dependency. This is the data that turns 'the site is slow' into 'calls to the pricing service are queueing', usually within a minute rather than an hour.
It also surfaces retry amplification, which is invisible from either side alone. If a caller retries three times and its own caller retries three times, one user action becomes nine requests to the service that is already struggling. Retry counts per dependency make that obvious; without them it looks like a sudden traffic spike with no source.
How do you keep the cost under control?
By controlling cardinality first and volume second. The cost of a metrics system is driven by the number of distinct label combinations, so putting a user id, order id, or full URL path into a metric label creates one time series per value and produces a bill nobody predicted. Identifiers belong in traces and logs, which are designed for them. Metric labels should hold service, endpoint, status class and a small number of other bounded values.
For traces, sample rather than record everything, but sample intelligently. Keeping a small fraction of successful requests and every error and every slow request gives most of the diagnostic value at a fraction of the volume. A flat low sampling rate quietly discards the rare failure you needed.
Logs are usually the largest line item and the easiest to reduce. Debug-level logging in production for every request is the common culprit, and moving detail into trace spans removes both the cost and the correlation problem at once. Short retention for high-volume logs and longer retention for errors and audit records is a reasonable default.
How do you know you are ready?
Take a real request identifier from a support ticket and try to reconstruct what happened to it across every service in under five minutes. If you can, you are ready to split further. If you cannot, adding services will make the next incident worse in a way no amount of architectural quality compensates for.
A second check is the alert list. Every alert should name a service, a symptom a user would notice, and the team that will be woken. Alerts on causes rather than symptoms, such as high CPU on one instance, generate noise and train people to ignore the page. If the list is long and mostly cause-based, it will not survive multiplication by the number of services you are about to create.
Both checks take an afternoon. Both are better predictors of whether a migration will go well than any review of the service boundaries themselves.
Common questions
- What observability do you need before adopting microservices?
- Trace context generated at the edge and propagated through every hop including message brokers, structured logs carrying the trace id, request rate and error rate and duration per endpoint recorded as distributions rather than averages, and health checks that distinguish starting from ready from alive. Without the first of these, no single customer request can be followed across services, which makes diagnosis guesswork.
- Why do distributed traces break at message queues?
- Because trace context travels in HTTP headers automatically but must be written into message headers deliberately. Teams instrument their HTTP calls, publish a message, and the trace ends there while a fresh unrelated trace begins in the consumer. Asynchronous hops are where the hardest failures occur, so propagating context into and out of broker messages matters more than the synchronous case, not less.
- Why measure dependency latency at the caller rather than the server?
- Because the two differ, and the gap is where outages live. A server reporting twenty milliseconds can appear to its caller as most of a second once connection pool waiting, DNS, TLS handshake, queueing and retries are counted, none of which the server sees. Client-side latency, error, timeout and retry counts per dependency also expose retry amplification, which is invisible from either side alone.
- What drives the cost of an observability platform?
- For metrics, label cardinality rather than traffic: each distinct combination of label values is a separate time series, so putting a user id, order id or raw URL path into a label multiplies cost without limit. Identifiers belong in traces and logs. For traces the driver is sampling and retention, and for logs it is raw volume, usually dominated by debug output left on in production.
- How should you sample distributed traces?
- Keep a small proportion of successful requests plus every error and every request slower than a set threshold. A flat low sampling rate is cheaper to configure and discards exactly the rare failures worth investigating. Sampling decisions that consider the outcome of the request give most of the diagnostic value of full capture at a small fraction of the storage.
