The notifications module at the fictional Magaliesberg Provincial Supplier Desk earned its extraction. For three tender closing weeks in a row, bursts of SMS and email messages had queued behind a slow messaging provider and held database connections that document uploads needed. The team ran the evidence gate from Part 1: the problem was named, measured and owned, the expected improvement was stated, and the review date was set. The steering committee approved a separate notifications service.
On the Monday after approval, Refilwe Dlamini, who carries the on-call phone, opened the code and searched for every call into the notifications module. There were fourteen. "Each of these used to be a function call," she said. "On Friday each one becomes a network call. What happens to the case if the notifications service is down?" Nobody had decided. The code would decide on the first bad day.
The Magaliesberg Provincial Supplier Desk is a fictional provincial supplier-support platform. Its people, systems and figures are illustrative. This part is about the work that starts the moment a module leaves the process.
What a network call can do that a function call cannot
Inside one process, a call to the notifications module either returned or threw an exception, immediately, and it ran inside the caller's database transaction when the caller wanted it to. After extraction, the same call has at least five new outcomes.
The last outcome, success that the caller never hears about, is the one that causes most of the damage. The caller times out, assumes failure and tries again, and the supplier receives the same "document rejected" SMS twice. Or the caller gives up, records the case as "notification failed", and the supplier received the message anyway.
None of these failure modes are new. Our articles on who owns the retry and why a successful payment can still end in a failed order cover retry storms, nested timeouts, sagas and idempotency keys in depth. What is new for the Supplier Desk is that fourteen calls that never needed any of that now all do, and they need it before Friday, not after the first incident.
A call conversion register
The team's answer was a register: one row per former in-process call, with a decision recorded for each column before the extraction was deployed. It is a small document, and it turns a vague worry into fourteen concrete decisions.
| Column | Question | Example answer for "notify supplier of rejected document" |
|---|---|---|
| Call | Which call, from which module? | Documents to Notifications: NotifyDocumentRejected |
| Kind | Query, command or event? | Event: Documents states a fact; it does not need an answer |
| Style | Synchronous or asynchronous? | Asynchronous through a queue |
| Must the caller wait? | Does the user journey depend on the result? | No. The rejection is saved whether or not the SMS is sent |
| Timeout | How long before the caller gives up? | Not applicable to the caller; the relay has a 5-second timeout per attempt |
| Retry owner | Which single layer retries, and how many times? | Two hops, one owner each: the outbox relay retries publishing to the queue; once published, the queue redelivers any message the notifications service has not acknowledged, then dead-letters it after one hour |
| Idempotency | What makes a repeat harmless? | A message key derived from document id and rejection version |
| Consistency | What if the caller commits and the call never happens? | The outbox row is written in the same transaction as the rejection |
| Failure visible to | Who sees it, and where? | The notifications service publishes NotificationSent or NotificationFailed; the case worker sees the status on the case; on-call is alerted when a message is dead-lettered |
| Trace | How is the call linked to the user's request? | Trace context is stored on the outbox row and sent with the message |
The register forced a useful realisation. Of the fourteen calls, eleven were really events: other modules telling notifications that something had happened. Only three needed an answer, and all three were queries for template previews in the administration screens. Treating every call as a synchronous request would have coupled document upload to the notification service's availability, which was exactly the problem the extraction was meant to remove.
Commands and events need different designs
The distinction between a query, a command and an event sounds academic until a network sits in the middle.
- A query asks for data and changes nothing. It can be retried safely. The design questions are timeout, caching and what to show if the answer does not arrive.
- A command asks another service to do something. If it is retried, the receiver must recognise the repeat, which means an idempotency key. The caller must also decide what to do if it never learns the outcome.
- An event states that something has already happened. The publisher does not wait for anyone. The design questions are delivery guarantees, ordering and how consumers handle duplicates.
The Supplier Desk moved eleven of the fourteen calls to events. The remaining three queries got short timeouts and a clear "preview unavailable" message for administrators.
The outbox: commit the fact and the message together
Moving to events created one more problem. When Documents rejects a document, it writes the rejection to its own schema and must also publish an event. If it writes the rejection and then crashes before publishing, the supplier is never told. If it publishes first and the database write then fails, the supplier is told about a rejection that does not exist.
The common answer is the transactional outbox pattern. AWS's prescriptive guidance (checked on 5 November 2026) describes it as resolving "the dual write operations issue", where one operation must update a database and send a message. The business change and an outbox row are written in the same database transaction. A separate relay reads committed outbox rows and publishes them.
The same guidance makes the consequence explicit: a relay can send a message more than once, so the consumer must be idempotent. At the Supplier Desk, the notifications service records each message key it has processed and ignores repeats. A duplicate event therefore almost always costs a database lookup, not a second SMS to a supplier. A crash between sending the SMS and recording the key can still produce a duplicate, unless the messaging provider accepts an idempotency key of its own, so the team checked which of its providers do.
Timeouts, retries and one owner
For the three remaining synchronous queries, the team applied the guidance in the AWS Builders' Library article Timeouts, retries and backoff with jitter (December 2019, checked on 5 November 2026). It recommends setting timeouts on every remote call, choosing values from the downstream service's measured latency, and using backoff with jitter and a limit on the number of retries. It also warns that retrying independently at every layer multiplies load, and gives an example of retries across five layers increasing database load 243 times.
The Supplier Desk's rule matches the one in our model gateway series: for every hop, exactly one layer owns the retry. An event crosses two hops. From the outbox to the queue, the relay owns retries. From the queue to the SMS send, the queue owns them: if the notifications service fails to send, it does not acknowledge the message, and the queue redelivers it with a delay until the one-hour limit, after which the message goes to a dead-letter queue and on-call is alerted. The relay never re-sends a message it has already published. For the preview queries, it is the HTTP client in the administration module, with one retry and a 2-second total budget. The notifications service itself does not retry calls to the messaging provider in a loop. It leaves the message unacknowledged and lets the queue bring it back.
Tracing has to cross the gap
Before extraction, one request produced one log stream and one stack trace. After extraction, a rejected document produces a request in the main application, an outbox row, a relay publish, a queue message, a notifications service handler and a call to the messaging provider. Without a shared identifier, those are six unrelated records.
The W3C Trace Context Recommendation (November 2021) defines a standard traceparent header for carrying that identifier between systems. The Supplier Desk stores the trace context on each outbox row, so that the asynchronous hop does not break the trace. Our article on logs, metrics and traces explains why that matters during an incident: most of the time is spent reconstructing the path.
Make partial failure visible to people
The last column of the register asks who sees a failure. That question changed the case screens. Before extraction, "notification sent" was assumed. Afterwards, the notifications service publishes NotificationSent and NotificationFailed events back to the main application, which updates a small, event-fed status copy, and every case shows the state of its notifications: pending, sent, failed and retrying, or abandoned after one hour. Case workers now know when to phone a supplier instead of assuming the SMS arrived.
This is the same principle as the order states in our saga article: if the system cannot represent a partial failure, people will discover it on their own, usually from an annoyed supplier.
A checklist before the first network call goes live
The checklist below is narrower than a full readiness review, which What does production-ready actually mean? covers, but it is the part most often skipped when a module becomes a service.
- Every former in-process call is in the register, with kind, style, timeout, retry owner, idempotency, consistency, visibility and trace decisions.
- Calls that are really events have been converted to events.
- Every command and event carries a key that makes a repeat harmless.
- Facts and their messages are committed together, through an outbox or equivalent.
- Exactly one layer retries on each hop, with backoff, jitter and a retry limit, and failures end in a dead-letter queue someone watches.
- Trace context crosses every hop, including queues.
- Users and operators can see partial failure, and alerts fire before suppliers notice.
Key points
- Extraction turns function calls into network calls that can be slow, fail, repeat or succeed silently.
- A call conversion register makes one explicit decision per former call before the service goes live.
- Many calls are really events. Converting them removes availability coupling.
- Use an outbox to commit a fact and its message together, and make every consumer idempotent.
- One layer owns each retry, trace context crosses every hop, and partial failure is visible to people.
Previous: Part 2, A modular monolith is not a folder structure. Next: Part 4, One database does not mean shared ownership. All parts are on the series page.
How CloudNala can help
CloudNala prepares teams for the first network boundary in a system. We build the call conversion register with you, separate queries, commands and events, design the outbox, idempotency and retry ownership for each path, and check that tracing and on-call are ready before the extraction is deployed, so that the first bad day is a planned one.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Architecture Complexity Review or write to us at consult@cloudnala.co.za