The fastest way to make a pipeline green is to run it again. Every engineer knows this, and most pipelines make it easy: a retry button, an automatic rerun on failure, a setting that keeps only the latest result. In a busy fortnight it feels like housekeeping. The test was flaky, the rerun passed, move on.
The cost is invisible until an incident. Then the post-incident review asks whether anyone saw this coming, and the honest answer is that the system did see it, twice, and the evidence was thrown away. A failed test is often the most informative record a delivery team produces. Deleting or overwriting it destroys learning, auditability and the trend that would have turned an intermittent nuisance into a defect someone fixed.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. Kudu's Kudu Repeat app submits medical aid claims through an external claims integration. The tests for that journey had been "a bit flaky" for months.
Two ways to record the same three runs
In the top lane, Kudu's original configuration. Run 1 fails with a timeout calling the claims integration. The pipeline retries. Run 2 passes. The record shows a single result: passed. Nothing about the first failure survives except a vague memory that "the claim tests are flaky".
In the bottom lane, the same three runs recorded append-only. Run 1 fails with a timeout. Run 2 fails with the same timeout. Run 3 passes. All three are kept. Across a month, the pattern is unmistakable: the claim API times out on roughly one run in five, always above a certain payload size. Someone raises defect KR-1131. The cause turns out to be a timeout setting that is too tight for larger claims, which production would have hit on a busy month-start morning.
Same tests, same infrastructure, same people. The only difference was whether the failures were kept.
Why rerun-to-green is so tempting
Flaky tests are a genuine problem. A suite where ten percent of runs fail for reasons unrelated to the code trains people to ignore red builds, which is worse than having no tests. Automatic reruns are a reasonable response to that. The mistake is not rerunning. It is discarding the evidence from the failed run and treating the final green as the only truth.
There is also a newer temptation. AI agents that maintain test suites are often asked to "fix flaky tests". An agent can do this in several ways, and not all of them are fixes. It can find and correct a real race condition. It can also add a longer wait, loosen an assertion, or mark the test as skipped. The last two make the suite greener and the evidence weaker. Without the history of failures, nobody can tell which kind of fix was made. Part 9 covers the guardrails that stop this.
What a failed result should keep
A useful failure record has six parts:
Run ID and build digest. So the exact failing combination can be reproduced, not approximately.
Environment and data set. Many failures are environmental: a test database that was reset, a dependency stub that was down. Recording the environment lets the team separate "the product is broken" from "the test environment is broken", and the second category is worth fixing too.
Logs, trace ID and screenshots. Enough to diagnose without rerunning. A trace ID that links to the same request in the application's own telemetry is especially valuable, because it connects the test failure to what the system actually did.
Failure classification. Product defect, test defect, data problem or environment problem. This is a judgement, usually suggested by tooling or an agent and confirmed by a person. The classification sits alongside the original record; it never replaces it.
Linked defect or waiver. What was done about it: a defect raised, a test fixed, an exception accepted. A failure with no follow-up link is an open question.
Retention period. How long the record is kept. For most teams, a year of full records and a longer summary is a sensible starting point. Where evidence supports regulated decisions, retention should follow the organisation's records policy.
Storage is not the obstacle
The usual objection is cost. Keeping every log and screenshot from every run sounds expensive. In practice it rarely is. Test artefacts compress well, object storage is cheap, and lifecycle rules can move older records to cheaper tiers automatically. For a team of Kudu's size, a year of full test evidence is measured in tens of gigabytes. The cost of one unexplained production incident is far larger.
The more real obstacle is tooling defaults. Many CI systems and test reporters are configured to show the latest result and discard or hide the rest. Changing that is usually a configuration setting and a storage bucket, not a project.
What the history makes possible
Once failures are kept, three useful practices follow.
Flakiness becomes measurable. Each test gets a failure rate across runs on unchanged code. Tests above a threshold are quarantined from release gates and assigned to someone, rather than silently retried forever.
Trends become visible. A journey whose failure rate is creeping up over several releases is a warning, even if every individual release went out green.
Incidents become explainable. When something breaks in production, the first question is whether the test evidence showed any sign of it. With append-only records, that question has an answer.
This connects directly to how an organisation treats evidence more generally. The prompt is part of the evidence argues that records of how an AI-assisted decision was made must be kept intact. Test evidence is the same principle applied to software: the record of what happened is not something to tidy up.
Classifying failures without losing them
Classification is where many teams quietly lose evidence even after they stop overwriting results. A failure is labelled "environment" and filtered out of every report, and the label is never checked. If the environment problem was really a product problem under different conditions, the label has hidden it as effectively as deletion would.
Three practices keep classification honest. First, a classification is always a separate field added to the record, with the name of whoever set it, so it can be revisited. Second, environment failures are counted and reported in their own right, because an unreliable test environment is a quality problem that wastes engineering time. Third, a sample of classifications is reviewed each month: pick a handful of failures labelled "environment" or "flaky" and check whether the label still holds. At Kudu, the first such review reclassified the claim API timeouts from "flaky" to "product defect", which is how KR-1131 came to exist.
AI tooling can propose classifications quickly, and it is often right about obvious environment failures such as a database that was not running. It should not have the last word on anything touching a high-risk requirement.
A simple rule
If a team adopts one rule from this part, it should be this: no test result is ever overwritten; a later result is added beside it. Everything else, from classification to quarantine to trend analysis, builds on that.
Next: Part 7, Offline-first assurance for intermittent environments. All parts are on the series page.
How CloudNala can help
We help teams move test evidence to an append-only model: reconfiguring pipelines and reporters, setting retention and storage rules, introducing failure classification and putting a flaky-test quarantine process in place. The result is a test history that explains incidents instead of hiding them.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za