A customer completes checkout. Their banking app shows the payment. A minute later they receive an email saying the order could not be processed. They phone the call centre, where the agent can see a failed order but no refund, and has no idea whether one is coming.
Most organisations treat this as an embarrassing edge case. It is better understood as a design question that was never answered: what should happen when a business action that spans several systems succeeds in some of them and fails in others?
Marula Commerce is fictional, and its systems and figures are illustrative. A Marula checkout touches four systems: the shop platform creates the order, an external payments provider captures the money, the warehouse system reserves stock, and a courier API books collection. None of these share a database. There is no single "commit" that makes all four succeed or fail together.
Why there is no simple transaction
Inside one database, a transaction gives an all-or-nothing guarantee. Debit one account, credit another, and if anything fails in between, the database rolls both back as if nothing happened.
Across systems owned by different parties, that guarantee does not exist in any practical form. The payments provider will not hold your customer's money in a half-committed state while it waits to hear whether your warehouse has stock. Techniques that coordinate a commit across several systems do exist, but they require every participant to support them, they block while waiting, and external providers almost never take part. So each step commits on its own, and the design has to say what happens when a later step fails.
A saga: every step has an undo
The usual answer is called a saga. It is a sequence of steps, each of which commits independently, where every step that can be undone has a matching compensation: an action that reverses its business effect.
In Marula's original design, shown above, the forward steps are create order, capture payment, reserve stock and book courier. The compensations are cancel order, refund payment, release stock and cancel booking. If stock reservation fails, the saga runs the compensations for the steps that already succeeded, newest first: refund the payment, then cancel the order. The courier was never booked, so there is nothing to cancel.
Two points about compensation are easy to miss.
A compensation is a new business action, not a rollback. A refund is not the payment "un-happening". It is a separate transaction that the customer will see on their statement, that may take days to settle, and that can itself fail. The design has to treat it with the same care as the original payment.
Some steps cannot be compensated cleanly. An email that has been sent cannot be unsent. A courier that has already collected the parcel cannot be uncollected by an API call. Those steps should come as late in the sequence as possible, after everything that is likely to fail.
That second point changed Marula's design. Originally, stock was reserved after payment because the payment step was the one the business cared most about. Reversing the order (reserve stock, then capture payment) means the most common failure, being out of stock, happens before any money moves. The refund path still exists, but it is now rare.
State the customer can see
The original Marula design had three order states: pending, confirmed and failed. "Failed" was used for every problem, including the case where money had been captured and not yet returned. That is where the angry phone calls came from.
The revised design adds a refund pending state, visible to the customer and the call centre, with an expected date. If the refund itself fails twice, the order moves to an operations queue for a person to resolve. The order only becomes "cancelled, refunded" when the payments provider confirms the refund.
These states are not cosmetic. They are how the business tells the truth about partial failure. If your order model cannot represent "we have your money and we are giving it back", your customers will discover that state on their own.
Idempotency: doing the same thing twice safely
The other half of the problem is retries. Networks fail. When Marula's order service calls the payments provider and the connection drops, it does not know whether the payment went through. If it simply tries again, the customer may be charged twice.
The defence is an idempotency key: a unique value the caller generates once per business action and sends with every attempt. The provider records the key with the first successful charge. A retry with the same key returns the original result instead of charging again. Most mature payment providers support this, and every internal API that changes something important should too.
The same principle applies to compensations. If the refund call is retried, it must carry its own idempotency key, or a nervous retry policy will refund the customer twice.
Orchestrated or choreographed?
There are two common ways to run a saga.
| Orchestration | Choreography | |
|---|---|---|
| How it works | One coordinator calls each step and decides what to do on failure | Each service reacts to events from the others |
| Strength | The whole flow is visible in one place | Services stay loosely coupled |
| Weakness | The coordinator becomes important and must be reliable | Hard to see the overall state; failures are harder to reason about |
| Good fit | Checkout, onboarding, anything with money or legal effect | Simple, loosely related reactions (send a thank-you email) |
For flows involving money, CloudNala's default is orchestration: one component that owns the order's state, records each step, and can be asked "where is order 9182 and why?". Durable workflow engines exist for exactly this purpose, and they also solve the problem of a coordinator crashing halfway through.
Questions to ask about any multi-system action
- Which steps commit independently, and in what order?
- What is the compensation for each step, and can it fail?
- Which steps cannot be undone, and are they last?
- Does every call that changes something carry an idempotency key?
- Which states can the customer and the support team see?
- Who resolves the cases that automation cannot?
If these questions do not have written answers, the system will still answer them in production. It will just do it inconsistently, and the customer will find out first.
Next: Part 4, Logs, metrics and traces. All parts are on the series page. For the same failure pattern in an AI agent's tool calls, see When the tool fails but the answer looks complete.
How CloudNala can help
We help teams map multi-system business actions, design the compensation and state model, and specify idempotency for every call that changes data. The output is a saga map per critical flow that engineers, operations and the call centre can all read.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request an Architecture Review or write to us at consult@cloudnala.co.za