How do you test a distributed system?
The instinct is to build one environment with every service in it and run the old end-to-end suite against it. That works for three services and degrades badly beyond that, because the suite can only be as stable as the least stable service in it and nobody owns a failure that spans four teams. The replacement is not one technique but a redistribution of where confidence comes from.
Why does the shared end-to-end environment stop working?
Because its reliability is the product of every service in it, and its failures have no owner. A suite that touches six services will fail regularly for reasons unrelated to the change under test, and the rational response to a suite that cries wolf is to rerun it. Once rerunning is normal, the suite has stopped being a gate and become a tax.
The second problem is version skew. The environment holds one version of each service, so a test either validates combinations that will never ship together or requires everyone to deploy in a particular order to be meaningful. Teams then start queueing for the environment, which removes the independent deployment that motivated the split.
This does not mean no end-to-end tests. It means a small number of them, covering the two or three journeys that make money, owned by a named team, and treated as production monitoring rather than as a pull request gate.
Where does confidence come from instead?
From tests that run inside one service's pipeline without needing anyone else's. That is the design constraint that makes the whole strategy work: if a test requires another team's service to be running, it cannot be part of your deploy pipeline without coupling your release to theirs.
In practice this means the bulk of coverage sits in unit tests and in integration tests that run against real infrastructure started as containers: your actual database engine, your actual broker, not an in-memory substitute that behaves differently under concurrency. Testing against a fake database and shipping to a real one is a common source of production-only defects, particularly around transactions, constraints and ordering.
External dependencies are stubbed at the boundary, and the accuracy of those stubs is the thing that contract tests exist to guarantee. Without contract tests, a stub is a record of what you assumed the other service did on the day you wrote it.
What does each kind of test actually catch?
The useful way to compare them is by what they catch that nothing cheaper catches, and by what they cannot catch at all.
| Test type | Catches | Cannot catch | Runs in |
|---|---|---|---|
| Unit | Logic errors, edge cases in your own rules | Anything about integration or wiring | Milliseconds, no dependencies |
| Integration with real infra | Schema, transaction and concurrency mistakes | Mistaken assumptions about other services | Your pipeline, using containers |
| Consumer contract | A provider breaking a field a consumer relies on | Whether the business logic is correct | Both pipelines, no shared environment |
| Asynchronous flow | Duplicate, out-of-order and failed message handling | Cross-service timing under real load | Your pipeline, with a real broker |
| Critical journey | Wiring, configuration and identity mistakes | Anything not on the two or three chosen paths | A staging environment, and against production |
What is contract testing and what does it miss?
A consumer records exactly what it needs from a provider: the request it sends, the fields it reads from the response, and the status codes it handles. The provider then replays those expectations against its own code in its own pipeline. If a change would break a consumer, the provider's build fails before the change is released, and neither team needed a shared environment to find out.
It is the single highest-value practice after the split, because it targets the exact failure the split introduces: a provider changing something a consumer depended on, discovered in production. It also documents which consumers exist and what they use, which is information that otherwise lives only in people's heads.
What it misses is semantics. A contract test confirms the provider still returns a status field; it cannot confirm the provider still means the same thing by the value 'active'. It also misses anything about load, latency or ordering. Contract tests replace most integration environments, not all end-to-end thinking.
How do you test asynchronous flows?
By testing each side separately against a real broker. Publish a message to the topic your consumer subscribes to and assert the state change it should produce, without the publishing service involved at all. Then, in the publisher's own tests, assert that the expected message was produced, ideally by checking the outbox rather than the broker.
Avoid sleeping. A test that waits two seconds and then asserts is both slow and flaky, and it gets slower as more are added. Poll for the expected condition with a short interval and a firm timeout, so a passing test is fast and a failing one is decisive.
Then test the cases the broker guarantees you will get: deliver the same message twice and assert nothing happens the second time, deliver messages out of order and assert the outcome is still correct, and make a message fail repeatedly to confirm it reaches the dead letter queue rather than blocking the partition. These three tests catch most asynchronous production incidents, and almost nobody writes them until after the first one.
What can only be found in production?
Real data shapes, real concurrency, real latency tails and real version combinations. No pre-production environment holds the customer whose account has eleven thousand line items, and that is the customer whose request times out. The practices that address this are deployment practices rather than test practices: release to a small percentage of traffic first, keep changes behind flags that can be turned off without a deploy, and run a synthetic version of your critical journey continuously against production so you learn about breakage before a customer reports it.
The other half is deliberately breaking things where you can watch. Add latency to a dependency in a staging environment and check what the caller does. Stop a dependency entirely and check whether the failure is contained or spreads. Most systems have never been observed in either condition, and their behaviour is usually not what the diagram suggests.
A test to run this afternoon: pick your most important service, point one of its dependencies at an address that accepts connections and never replies, and watch what a user sees. If the answer is a hung page or a cascade into unrelated features, you have found the highest-value fix available to you, and it is a timeout.
Common questions
- Why do end-to-end tests fail in a microservices architecture?
- Their reliability is the product of every service involved, so a suite spanning six services fails regularly for reasons unrelated to the change under test, and nobody owns a failure that crosses four teams. Once rerunning a flaky suite is routine, it has stopped being a gate. Keep a small number covering the journeys that make money, owned by a named team, and get confidence elsewhere.
- What is consumer-driven contract testing?
- The consumer records exactly what it needs from a provider: the request it sends, the response fields it reads and the status codes it handles. The provider replays those expectations in its own pipeline, so a breaking change fails the provider's build before release. Neither team needs a shared environment, and the contracts also document which consumers exist and what each one actually uses.
- What do contract tests not catch?
- Semantics, performance and ordering. A contract test confirms a provider still returns a status field but not that the value 'active' still means what the consumer assumes. It says nothing about latency under load, nothing about message ordering, and nothing about whether the business logic behind the interface is correct. It replaces most integration environments rather than all end-to-end thinking.
- How do you test asynchronous message flows?
- Test each side separately against a real broker. Publish a message to the consumer's topic and assert the state change, with no publisher involved; in the publisher's tests, assert the message was produced, usually by checking the outbox. Poll for conditions rather than sleeping. Then explicitly test duplicate delivery, out-of-order delivery and a message that always fails reaching the dead letter queue.
- Should you use in-memory databases for testing services?
- Prefer the real engine started as a container. In-memory substitutes differ from production databases in exactly the areas that cause production defects: transaction behaviour, constraint enforcement, locking and ordering under concurrency. Container startup adds seconds to a pipeline and removes a class of bugs that otherwise appears only after deployment.
