Stateless services: where does the state actually go?
Stateless does not mean the application has no state. It means no single instance holds state that anything else depends on, so any instance can answer any request and any instance can be destroyed without consequence. The state still exists; it has moved somewhere designed to keep it. Teams usually move sessions, declare the job done, and leave four other kinds of state exactly where they were.
What does stateless actually mean?
That any instance can serve any request without needing to have seen a previous one. The test is not philosophical: send two consecutive requests from the same user to two different instances and see whether the second one works. If it does, the service is stateless with respect to that flow. If the user is logged out, or the upload has vanished, or the wizard has forgotten step one, it is not.
The state has not been eliminated by making a service stateless, only relocated to something whose job is durability: a database, a cache cluster, object storage, a message broker. Those are the stateful parts of the system and they are deliberately few, because state is the expensive thing to run in multiples.
This is why the phrase misleads. The goal is not an application without memory. It is an application where memory is external, shared and explicitly managed, so that instances become interchangeable and disposable.
What kinds of state are hiding in a typical application?
Six, and only the first is usually addressed. Sessions and authentication, uploaded and generated files, caches, scheduled work and other singletons, long-lived client connections, and partially completed work in flight when the process is killed.
Each has a different destination and a different failure signature, and the failure signatures matter more than the list because they are how you find them in an existing codebase. A duplicate nightly email is a scheduler problem. A file that downloads on one refresh and 404s on the next is a local disk problem. Both look like flakiness until you know what to look for.
| Kind of state | Where it belongs | How it fails if left in the instance |
|---|---|---|
| Sessions and login | Signed cookie, or a shared cache such as Redis | Users logged out at random as the load balancer moves them |
| Uploads and generated files | Object storage, referenced by key | Files present on one instance and missing on every other |
| Caches | Shared cache, or per-instance but treated as disposable | Two instances disagreeing about the same value indefinitely |
| Scheduled jobs and timers | External scheduler, or leader election, or a database lock | The nightly job running once per instance every night |
| Websockets and long-lived connections | A dedicated gateway tier plus a pub/sub backplane | Messages delivered only to users attached to the sending instance |
| Work in flight during shutdown | A queue with acknowledgement, plus idempotent handlers | Requests dropped and jobs half-done on every deploy |
Where should sessions go?
Either into a signed cookie held by the client, or into a shared store that every instance can read. Both are correct, and they trade the same thing against each other: the cookie approach needs no infrastructure and cannot be revoked before expiry, while the shared store can revoke instantly and adds a dependency that must be running for anyone to log in.
The revocation difference is the one that decides it. If your product needs an administrator to be able to end a session immediately, when someone is dismissed or a device is lost, self-contained tokens will not do it without adding a check against a deny list, at which point you have the shared store anyway and a more complicated design than starting there.
Keep the payload small either way. Cookies are sent on every request, so anything you put in one is paid for on every request forever, and the temptation to store user preferences or permissions in the token is how a token grows past the size where it is efficient.
What happens to scheduled jobs when you run three instances?
They run three times. This is the most common and most damaging state bug in a container migration, because an in-process timer is invisible in code review and behaves perfectly on one instance. It only misbehaves in production, and it misbehaves by doing real work repeatedly: sending the same invoice reminder, charging the same subscription, or writing the same report three times with slightly different timestamps.
There are three standard fixes. Move the schedule out of the application, into a platform cron object that starts a one-off job. Elect a leader, so exactly one instance considers itself responsible. Or take an advisory lock in the database at the start of the task and skip if it is already held, which is the smallest change and works well for tasks that run infrequently.
Whichever you pick, the task itself should also be idempotent, because leader election and locks both have edge cases around network partitions where two instances briefly believe they hold it. If running the job twice produces the same result as running it once, none of those edge cases become an incident.
Are sticky sessions ever acceptable?
As a deliberate, dated migration step, yes. Configuring the load balancer to pin a user to one instance makes a stateful application survive horizontal scaling, and if the alternative is delaying a datacentre exit by two quarters, it is the correct engineering decision.
The costs are worth stating so the decision is made with open eyes. Deploys become disruptive because every instance replacement evicts whatever its users were holding. Scaling down does the same. Load distribution becomes uneven, since users are not uniform and one instance will collect the heavy ones. And the underlying problem stays invisible, because the symptom that would have found it has been suppressed.
Set a date and a condition when you turn it on, and record what it is compensating for. Sticky sessions applied as a permanent solution are how an application arrives at its fifth year on a container platform still unable to survive a rolling deploy.
What about workloads that genuinely are stateful?
Some things hold state as their entire purpose: databases, message brokers, search indexes, stream processors with local windows, and gateways holding thousands of open connections. These are not badly designed applications waiting to be fixed. The right approach is to concentrate the statefulness in as few components as possible and treat them differently from everything else.
For most organisations that means buying them. A managed database handles failover, backup, patching and replica promotion, and doing those things correctly yourself is a specialism rather than a task. Running your own is defensible when a specific requirement demands it, and it should be a decision someone made rather than a default that arrived with a Helm chart.
When you do run stateful workloads on an orchestrator, they need the parts of the platform designed for them: stable identities, persistent volumes that follow the instance, ordered startup and shutdown, and a tested restore. The last one is the one that gets skipped, and a backup nobody has restored is a hypothesis rather than a backup.
Common questions
- What does stateless mean for a web service?
- That any instance can serve any request without having seen a previous one, so instances are interchangeable and disposable. The state still exists, but it lives in a database, shared cache, object store or message broker rather than inside a process. The practical test is to send two consecutive requests from one user to two different instances and check that the second still works.
- Where should session data be stored in a containerised app?
- Either in a signed cookie held by the client or in a shared cache every instance can read. Cookies need no extra infrastructure but cannot be revoked before they expire. A shared store allows instant revocation at the cost of a dependency that must be running for anyone to log in. If administrators must be able to end a session immediately, use the shared store.
- Why do scheduled jobs run multiple times when you scale out?
- Because an in-process timer exists inside every instance, so three replicas mean three executions. It passes code review and works on one instance, then sends duplicate invoices in production. Fix it by moving the schedule to a platform cron job, electing a single leader, or taking a database advisory lock at the start of the task, and make the task idempotent regardless.
- Are sticky sessions a valid solution?
- As a dated migration step, yes. Pinning users to one instance lets a stateful application survive horizontal scaling and can be the right call against a deadline. The costs are that every deploy and every scale-down evicts users, load distributes unevenly, and the underlying problem stays hidden because its symptom has been suppressed. Record what it is compensating for and set a date to remove it.
- Can stateful applications run on Kubernetes?
- Yes, using the parts of the platform built for it: stable network identities, persistent volumes that follow the instance, and ordered startup and shutdown. The harder question is whether you should. Databases and brokers need failover, patching and tested restores, which is a specialism rather than a task, so most organisations are better served buying the managed version and concentrating statefulness in as few components as possible.
