On a Saturday morning in April, Kudu's store system lost its connection to the central prescription service for about two hours. The app's status lookups fell back to a cached copy taken just before the outage. During those two hours, stores pulled back eleven requests that had already been marked ready: a stock count was wrong in one store and a batch of labels had been misprinted in another. The central service recorded the change. The cache still said "ready for collection", and the assistant told those patients so, cheerfully and accurately according to its data. Several drove to a store and waited. Nobody was harmed, and no request held for pharmacist review was affected. It was still an incident, and the review was thorough.
The review produced five actions: add a timestamp to cached status, alert on stale caches, update the assistant's instructions, brief store staff and tell the courier partner. All five were done within a fortnight. Then the release manager asked what would stop the same failure returning after the next refactoring. Had a developer later removed the timestamp field because nothing appeared to use it, nothing in the test suite would have noticed, because none of the five actions was a test.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. This final part of the series closes the loop that Part 10 and Part 11 opened: how what goes wrong in production becomes evidence that it will not go wrong again.
Why actions decay and tests do not
Google's site reliability engineering book describes a blameless postmortem as one that focuses on "identifying the contributing causes of the incident" and records "the follow-up actions to prevent the incident from recurring" (Postmortem Culture). That is the right starting point. The weakness is in the word actions. An action is done once. A system keeps changing after it is done.
A regression test is different. It encodes the lesson in a form that runs on every release, fails when the lesson is forgotten and names the incident it came from. It survives staff turnover, refactoring and model upgrades. For AI features, where a prompt edit or model change can undo a fix without touching any code, that durability matters more than usual.
The rule Kudu adopted is simple: every incident review must produce at least one test case, or a written reason why no test could detect a recurrence. The reason is rare and is reviewed by the release manager.
Evals that gate real work already argues for building evaluation sets from real failures. This part adds the mechanics that make the habit stick: each contributing cause mapped to its own test layer, cases reproduced with synthetic data, and a record that explains why each case exists.
From one incident to several cases
An incident rarely maps to a single test, because it rarely has a single cause. The Saturday incident had three contributing causes, and each needed a test at a different layer.
| Contributing cause | Layer | Test case |
|---|---|---|
| The status service returned cached data with no age indicator | Tool contract | Assertion: every status response carries a last-updated timestamp; the test fails if the field is missing or null |
| The assistant treated any status as current | Required behaviour | Labelled cases: when status is older than the agreed limit, the assistant states when it was last updated and does not confirm readiness |
| Nobody knew the cache was stale | Monitoring | Live check: alert when cached status is served for more than ten minutes; tested by injecting a stale cache in the test environment |
The first case is the one that would catch a refactoring that quietly drops the field. It is a deterministic assertion that costs nothing to run. The second goes into the labelled required-behaviour set from Part 10, so it runs on every model or prompt change. The third tests the alert itself, because a monitor that has never fired in a test may not fire in production.
The layer choice follows the routing from Part 10: put each case in the cheapest layer that can detect a recurrence of its cause. A cause in the tool contract does not need a model-graded rubric. A cause in the assistant's behaviour cannot be caught by an assertion on the tool.
Near misses and complaints count too
Waiting for incidents is a slow way to learn. Kudu feeds three other sources into the same process:
- Near misses. A pharmacist notices the assistant almost gave a patient the wrong collection time before a check stopped it. Nothing reached the patient. The check worked, which is exactly why a case is needed: if the check is ever weakened, the near miss becomes an incident.
- Complaints and corrections. When a patient or store corrects something the assistant said, the correction is reviewed weekly. Repeated patterns become cases.
- Red-team findings. Every successful attack from Part 11 is already a case by design.
The question for each is the same: what would a test need to check to know this cannot happen silently again?
Reproduce without the patient
Incident evidence at a pharmacy contains health information. The transcript of the Saturday conversations included patient names and medicine names. Under POPIA, health information is special personal information, as our plain-language guide to which data AI can touch explains. Copying those transcripts into a test suite that every developer and every test agent can read would create a new privacy problem to fix an old reliability one. This is general information, not legal advice; Kudu's privacy owner decides what is permitted.
Kudu's rule is that cases are reproduced, not copied. The test author keeps the shape of the incident and replaces every personal detail with synthetic data: a synthetic patient, a synthetic prescription, a timestamp and a status that recreate the conditions. The original evidence stays in the incident record under its own access controls, and the test case links to that record by identifier. A reviewer who needs the original can follow the link if they are entitled to. The suite itself stays synthetic, which is consistent with the data tiers in Part 9.
A case needs a record, not only code
A test case that says only "check timestamp present" will be deleted by the next person who does not understand why it is there. Kudu gives every incident-derived case a short record.
- Origin. The incident, near miss, complaint or red-team finding it came from, by identifier.
- Cause. The contributing cause it guards against, in one sentence.
- Requirement. The requirement or definition-of-good statement it protects, so it joins the chain from Part 3.
- Layer and routing. Assertion, behaviour set, rubric, adversarial or monitoring, and which change types trigger it.
- Owner. The person who maintains it and decides whether it is still needed.
- Review date. When someone will check that it still tests something real.
The record is what lets the suite be pruned safely. A case can be retired when its cause is designed out completely, for example when the cache is removed altogether, and the record shows why it is no longer needed. Without a record, cases are either kept forever or deleted by guesswork. Either way the suite stops being evidence. When an incident-derived case fails, its failure is kept like any other, as Part 6 requires.
Keep the suite honest as it grows
An incident-driven suite grows steadily, and a growing suite can become slow, noisy and ignored. Three habits keep it useful.
- Deduplicate at creation. Before adding a case, check whether an existing case already guards the cause. Strengthen that one instead.
- Review quarterly. The release manager and the case owners look at cases that have never failed, cases that fail intermittently, and cases whose review date has passed. Flaky cases are fixed or removed, never silently quarantined.
- Report lineage, not only counts. The release record states how many incident-derived cases ran and passed, and lists any added since the last release with their origin. That tells a reader that the system is learning, not just that it has many tests.
In October, six months after the Saturday incident, a developer refactoring the status service removed the timestamp field because nothing appeared to use it. This time the assertion failed in the pull request. The developer read the case record, saw the incident it came from and restored the field. The whole episode took ten minutes and never reached a patient. That is the return on the habit.
Bringing the series together
Release on Evidence began with a simple claim: a release decision is only as good as the chain of evidence behind it. The first nine parts built that chain for conventional software and AI-assisted testing. The last three extended it to AI features themselves. Part 10 required a written definition of good and routed each change to the evaluation that can see its likely failure. Part 11 tested what an agent can be made to do and judged it by state. This part closed the loop, so that what production teaches becomes part of the evidence for every release after it.
Key points
- Incident actions are done once and decay; regression tests run on every release and fail when a lesson is forgotten.
- Require every incident review to produce at least one test case, or a reviewed reason why none can detect a recurrence.
- Map each contributing cause to the cheapest layer that can detect it: tool assertion, behaviour set, rubric, adversarial case or monitoring check.
- Feed near misses, complaints and red-team findings into the same process, and reproduce cases with synthetic data rather than copying personal information.
- Give every case a record with origin, cause, requirement, layer, owner and review date, and report lineage in the release record.
This is the final part of Release on Evidence. Start again from Part 1, A test without a requirement is orphan evidence. For the incident process itself, see Classify the incident before you argue and AI-assisted incident response.
How CloudNala can help
CloudNala helps teams connect incident management to release evidence. We review your recent incidents and near misses, turn their contributing causes into test cases at the right layer, set up synthetic reproduction and case records that respect your privacy obligations, and design the quarterly review that keeps the suite small enough to trust.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za