Most release evidence is produced in a data centre or a cloud region with a reliable connection. The test runner talks to the application, the results land in a central store, the dashboard updates. That model breaks down as soon as part of the system runs somewhere less predictable: a store, a clinic, a depot, a branch, a vehicle.
In South Africa this is not an edge case. Retail chains, pharmacy groups, logistics firms, municipalities and health services all run software on devices and in sites where connectivity is intermittent, whether because of local network quality, power interruptions or simply distance. When evidence has to travel over an unreliable link, the design has to assume it will sometimes be late, and must never treat a missing result as a passing one.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. Each Kudu store has a tablet that pharmacists use to work the repeat-prescription hold queue. After every release, a set of smoke checks runs on the store tablets to confirm the hold queue, approvals and printing work locally.
The failure Kudu nearly missed
After an earlier release, Kudu's central dashboard showed smoke checks passing in 171 of 180 stores, with nine "not yet reported". The release was considered healthy, and the nine stores were assumed to be offline and fine.
Two days later, a regional manager mentioned that pharmacists in three of those stores had been approving held requests on paper because the tablet queue would not load. The smoke checks in those stores had run and failed, but the results had been held in a local buffer that was cleared when the tablets restarted during a power interruption. The central system never heard about the failures. It also never asked.
Nothing was wrong with the tests. The evidence simply did not survive the journey.
Store and forward
The fix is a well-established pattern from messaging systems, applied to evidence: store and forward.
Capture locally and seal it. When a smoke check runs on a store tablet, the result, logs and a timestamp are written to a local queue that survives restarts. Each record is hashed so it cannot be quietly changed later. The tablet's clock is recorded alongside, because device clocks drift and the central store will need to reconcile them.
Forward when possible. A small sync process sends queued records to the central evidence store whenever a connection is available. It retries with back-off and uses an identifier on each record, so a record sent twice is stored once.
Store centrally, append-only. The central store follows the rule from Part 6: nothing is overwritten. Late records are added with both their capture time and their arrival time.
Detect silence. This is the step Kudu was missing. The central store expects a result from every store after every release. A store that has not reported within a set window, 48 hours in Kudu's case, is flagged as unknown and followed up, not counted as passing and not ignored.
Connectivity states
It helps to be explicit about how capture behaves in each connectivity state.
Online. Results stream to the central store immediately.
Degraded. The link works but is slow or expensive. Results are batched, for example every fifteen minutes, to avoid competing with business traffic such as claim submissions.
Offline. Results queue locally. The queue is sized for several days and is durable across restarts and power loss. If the queue fills, the oldest low-priority records are summarised rather than dropped silently, and the overflow itself is recorded.
Resyncing. When the connection returns, queued records upload, hashes are verified and duplicates removed. Any gap, such as a period with no records at all, is reported explicitly rather than smoothed over.
The rule across all four states is the same: every result is kept, sealed and eventually forwarded, and every gap is visible.
Designing for power interruptions
Power interruptions deserve specific attention because they combine two problems: the device loses power, and the network often goes with it. Three practical choices make evidence capture more robust:
- Write evidence to durable storage before reporting success locally. A result that exists only in memory does not survive a restart.
- Record a heartbeat on start-up. When a device restarts, it writes a record saying so. A gap in heartbeats explains a gap in evidence.
- Keep the local evidence store small and simple. An append-only file or a lightweight local database is easier to recover after an unclean shutdown than something elaborate.
These are not exotic techniques. They are the same disciplines that make offline-capable point-of-sale and field-service apps reliable, applied to the evidence about those apps.
What changes for the release decision
Intermittent environments change how a release owner should read evidence.
"Not yet reported" is a status, not a pass. The release record should show three numbers: passed, failed and unknown. Unknown stays unknown until the evidence arrives or someone checks.
The decision may need a staged rollout. Where sites report slowly, it is often safer to release to a subset first, wait for their evidence, and then widen. Kudu now releases tablet changes to twenty stores, waits for complete evidence from all twenty, and then rolls out to the rest.
Late failures must reach someone. A failure that arrives three days after release still matters. The resync process should route late failures to the on-call owner, not just add them to a dashboard nobody is watching.
Keeping evidence trustworthy at the edge
Evidence captured on a device in a store raises a question central evidence does not: how does the central store know a record is genuine and unchanged? A tablet that has been offline for two days could, in principle, have had its local queue edited.
For most organisations, proportionate controls are enough. Each record carries a hash, and records are chained so that removing or altering one breaks the chain. The device signs records with a key issued to it, so the central store can tell which device produced which record. The central store rejects records that fail verification and reports them, rather than silently dropping them. None of this needs specialised hardware. It does need the evidence format to be designed with verification in mind from the start, because adding integrity later means distrusting everything already captured.
It is worth being clear about the limits. These controls show that a record was not altered after capture. They do not prove the test itself was meaningful, which is still the job of the evidence chain from Part 1.
Beyond stores
The same pattern applies to any environment where evidence capture is separated from evidence storage by an unreliable link: field devices, vehicles, rural clinics, edge sites, even developer laptops working on poor home connections. It also applies, in a different form, to integrations with client systems that are not always available. Evidence from a test against a third-party system that was down is not a pass or a fail. It is a known gap, and the record should say so.
For the network design side of this problem, see Network resilience, observability and cost.
Next: Part 8, Release readiness scores need an explanation. All parts are on the series page.
How CloudNala can help
We help organisations that run software in stores, clinics, depots and other intermittently connected sites design evidence capture that survives outages: local sealing, store-and-forward sync, silence detection and staged rollouts. The aim is a release record that is honest about what is known, what failed and what has not been heard from yet.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za