Distributed transactions and sagas
The cheapest solution to a distributed transaction is not having one. If two writes must both succeed or both fail, that is strong evidence they belong in the same service, and moving the boundary costs less than every alternative on this page. Sagas are what you use when the split is genuinely justified, and they are a business design problem wearing engineering clothes.
Why not just use a distributed transaction?
Two-phase commit exists and works, and it is still the wrong tool for services. It requires a coordinator that every participant trusts, and it holds locks in each database from the prepare phase until the commit, across a network, for the duration of the slowest participant. A coordinator failure between prepare and commit leaves participants holding locks with no authority to resolve them.
It is also unavailable in practice. Most of what a modern service talks to does not support it: HTTP APIs, message brokers, cloud object storage, third-party payment providers. A transaction that spans your database and someone else's API is not a technical possibility, so the pattern cannot cover the cases you most want covered.
Which leaves the honest first question. If two writes must be atomic, why are they in different services? A boundary that cuts through a genuine atomic requirement was drawn in the wrong place, and merging the two services back is a legitimate and often correct answer.
What is a saga?
A sequence of local transactions where each step is committed immediately, and each step has a compensating action that runs if a later step fails. There is no rollback, because the earlier writes are already visible to everyone. Instead there is a second business action that undoes the effect.
That distinction matters more than any code. Compensation is a new fact, not an erasure. You do not un-charge a card, you issue a refund, and the refund appears on the statement. You do not un-reserve stock in silence, you release the reservation, which may already have been visible to a customer who saw the item as unavailable. Each compensating action needs a business owner who agrees to what it does.
The other property to accept early is that sagas have no isolation. Between the first step and the last, other parts of the system can see a half-finished state: an order that exists with no payment, stock reserved for an order that will not complete. Somebody has to decide what those states are called and who is allowed to see them.
Which coordination approach should you pick?
Four options, in rough order of preference for most systems. The first one is not a saga at all, and it should be seriously considered before the others.
| Approach | How it works | Use when | Cost |
|---|---|---|---|
| Move the boundary | Both writes in one service, one local transaction | The two writes are genuinely atomic to the business | A larger service, which is usually acceptable |
| Choreographed saga | Each service reacts to the previous service's event | Two or three steps, stable flow | No single place describes the process, hard to debug |
| Orchestrated saga | One coordinator issues commands and tracks state | Four or more steps, or steps that change often | A component that must itself be durable and observable |
| Two-phase commit | Coordinator locks all participants, then commits | Two databases you control, inside one trust boundary | Locks held across the network, unsupported by most APIs |
What is the dual write problem?
Writing to your database and publishing a message are two separate operations with no shared transaction, so one can succeed while the other fails. If the database commit succeeds and the publish fails, the rest of the system never learns what happened. If you publish first and the commit fails, you have announced something that did not occur. Both happen regularly under normal restarts and network faults.
The standard fix is the transactional outbox. The service writes the message into an outbox table in the same local transaction as the state change, so either both are committed or neither is. A separate relay process, or change data capture reading the database log, then publishes from that table and marks rows as sent. Publication becomes at-least-once, which is why consumers must be idempotent anyway.
This is unavoidable rather than optional. Any service that changes state and tells other services about it has this problem, and a system without an outbox or an equivalent is silently losing events at a rate proportional to how often it restarts.
What happens when a step cannot be undone?
You order the steps so that irreversible actions happen last. An email cannot be unsent, a payout to a third party cannot be recalled, a parcel handed to a courier cannot be intercepted. Once a saga has taken such a step, the only remaining direction is forward, and the design must ensure everything that could fail has already been tried.
The technique is to split each risky step into a reservation and a confirmation. Authorise the card first and capture later. Reserve the stock, then commit it at dispatch. Compose the email, then send it after the last reversible step succeeds. The reservation is compensatable, the confirmation is not, and all reservations are taken before any confirmation is made.
Where an irreversible action genuinely must go first, the compensation is a human process, and it should be designed as one: an alert, a queue, and a named team that handles the exceptions. Pretending an automated compensation exists when it does not is how a system ends up quietly refunding nobody.
What does a saga cost you operationally?
A saga that is stuck halfway is a normal occurrence, so you need to see it. That means persisting the saga state with the step it reached, a timeout per step so a missing reply is detected rather than waited on forever, an alert on sagas older than their expected duration, and a way for an operator to retry or compensate a specific instance by hand.
It also means customer-facing consequences that are not engineering decisions. Support will see orders in intermediate states and needs language for them. A user may receive a confirmation for something that later fails, so somebody must decide whether to confirm optimistically or wait for the whole sequence. These conversations are much cheaper before launch.
The test worth running now: take your most important multi-step flow and write down, for each step, what the customer sees if the process stops there permanently. If any of those answers is unknown or embarrassing, the saga is not designed yet, no matter how much of the code exists.
Common questions
- What is a saga in microservices?
- A sequence of local transactions in different services, each committed immediately, where every step has a compensating action that runs if a later step fails. There is no rollback because earlier writes are already visible. Compensation is a new business action rather than an erasure: a refund rather than an un-charge, which is why each compensating step needs an owner who agrees to what it does.
- Why is two-phase commit a bad fit for microservices?
- It holds locks in every participating database from prepare until commit, across a network, for as long as the slowest participant takes, and a coordinator failure between the phases leaves participants stuck. It is also unsupported by most of what a service talks to, including HTTP APIs, message brokers and third-party payment providers, so it cannot cover the cases that most need covering.
- What is the transactional outbox pattern?
- A way to make a state change and its notification atomic. The service writes the outgoing message into an outbox table in the same database transaction as the state change, so both commit or neither does. A relay process or change data capture then publishes from that table and marks rows sent. It solves the dual write problem, where a database commit succeeds but the message publish fails.
- What happens if a saga step cannot be compensated?
- Irreversible steps such as sending email, paying a third party or handing a parcel to a courier are ordered last, after everything that can fail has already succeeded. Risky steps are split into a reservation and a confirmation, with all reservations taken before any confirmation. Where an irreversible action must come first, the compensation is a human process and needs an alert, a queue and a named team.
- How do you monitor sagas in production?
- Persist the saga state including the step it reached, set a timeout per step so a missing reply is detected rather than awaited indefinitely, alert on instances older than their expected duration, and provide an operator with a way to retry or compensate a single instance by hand. Stuck sagas are routine rather than exceptional, so the tooling to see and resolve them is part of the feature.
