Health checks and graceful shutdown

Why do deploys drop requests when nothing is broken?

Because the removal of an instance from routing and the instruction to shut down are sent in parallel, not in sequence. The orchestrator sends SIGTERM to your process at the same moment it updates the endpoint list, and that endpoint update then has to propagate to every load balancer, proxy and node in the path. For a short window, traffic is still being sent to a process that has begun exiting.

The window is small and the error count is proportional to traffic, which is why this is invisible on a quiet service and obvious on a busy one. It is also why it tends to be discovered after growth rather than at launch, and gets misattributed to whatever else changed at the same time.

Nothing in the platform can fix this for you, because only your application knows when it has finished. The correct behaviour has to be implemented in the process: stop declaring yourself ready, keep serving for a moment anyway, then drain and exit.

What is the difference between liveness and readiness?

They answer different questions and have opposite failure consequences. Readiness asks whether this instance should receive traffic right now; failing it removes the instance from the load balancer and nothing else. Liveness asks whether this process is beyond saving; failing it kills and restarts the container. Wiring the same handler to both is the most common configuration error in Kubernetes, and it converts a transient dependency blip into a restart loop.

The rule that prevents most trouble: readiness may consider things outside the process, liveness may not. If liveness checks a database, then a database outage restarts every instance you have, simultaneously, which turns a recoverable incident into an unrecoverable one because nothing is left running to serve cached responses or to reconnect when the database returns.

What should a readiness check actually check?

Only the things that would make this specific instance unable to answer a request that a different instance could answer. In practice that is a very short list: initialisation finished, required configuration present, and the process not currently shutting down. That is often the whole of it.

The tempting mistake is to have readiness verify every downstream dependency, which feels thorough and is actively dangerous. If every instance checks the same database, then a slow database fails readiness everywhere at once, the platform removes every instance from routing, and the outage escalates from degraded responses to no responses at all. The instances were fine. The check removed them.

There is a narrow exception. If an instance holds a connection that only it can repair, and it genuinely cannot serve without it, readiness is the right place to say so. The test to apply is whether a different instance would succeed at this request. If yes, fail readiness. If no other instance would do better, do not, because removing yourself helps nobody and reduces capacity during the exact period when capacity matters.

What does a correct shutdown sequence look like?

Six steps in a fixed order. Receive SIGTERM. Immediately start failing the readiness probe, which begins the removal from routing. Keep accepting and serving new requests as normal for a few seconds while that removal propagates. Then stop accepting new work. Finish requests already in progress and acknowledge or return any queued items you have claimed. Close database pools and broker connections. Exit.

The step that is nearly always missing is the third one. Teams handle SIGTERM by immediately refusing connections, which is intuitive and wrong: routing has not caught up yet, so refusing produces exactly the errors the handler was written to prevent. Continuing to serve during the propagation window is what makes the deploy clean.

The whole sequence has to complete inside the grace period the platform allows, after which the process is killed outright. Set that period longer than your slowest legitimate request, and if you have a request that takes several minutes, either shorten it or move it to a queue, because otherwise every deploy will terminate it mid-flight.

Why is there a sleep in the middle of everyone's shutdown handler?

Because endpoint removal is eventually consistent and nothing tells your process when it has finished. The orchestrator updates its record, controllers watch for the change, proxies on each node rewrite their rules, and external load balancers refresh on their own schedule. Your application has no way to observe the end of that chain, so it waits.

That is what the pre-stop hook or the deliberate delay after failing readiness is doing. It is not a workaround for a bug: it is the only mechanism available for a distributed system where the component being removed cannot see the removal complete. A few seconds is usually enough inside a cluster, and longer if there is an external load balancer with a slower refresh in the path.

Tune it by measurement rather than by copying a number. Run a load test, deploy during it, and increase the delay until the error count reaches zero. If it never reaches zero, the problem is elsewhere in the sequence, most often that new connections are being refused before the delay rather than after it.

How do you prove it works?

Generate steady traffic against a staging environment, at a rate high enough that a one-second gap would show, and count non-2xx responses. Then perform a rolling deploy while the load is running. A correct configuration produces zero failed requests. Any other result is a number, which is far more useful than an opinion about whether the handler looks right.

Repeat the test for the other three ways an instance disappears, because they behave differently: scaling down after a spike, a node being drained for maintenance, and an instance being evicted under memory pressure. The last one gives you no grace period at all, which is the case that tells you whether in-flight work was genuinely recoverable or merely usually fine.

Once it passes, keep the test. This behaviour regresses quietly, because the change that breaks it is usually an unrelated addition to startup or a new dependency opened at boot, and nothing else in the pipeline will notice.