A test suite can be enormous, green and almost useless at the moment it matters most. The release meeting asks a simple question: is this build safe to put in front of customers? The test report answers a different one: how many checks ran, and how many of them passed. Those are not the same question, and the gap between them is where most release risk hides.
The gap is getting wider. AI coding agents and AI test generators can now produce test cases faster than any team can read them. A product team that once had four hundred automated tests can have four thousand by the end of a quarter. The count goes up, the pass rate stays high, and the release owner is no better informed than before. In some ways they are worse informed, because a large number looks like assurance.
CloudNala's view is straightforward. A test only becomes evidence when it is connected to something the business asked for and to the decision that relied on it. A passing test with no requirement behind it is orphan evidence: it proves that some code did something, but nobody can say whether that something mattered, which risk it reduced or which release it supported. The fix is not more testing. It is a traceable chain from intent to release decision, and it is cheaper to build than most teams assume.
This series, Release on Evidence, works through that chain over nine parts. It starts here with the chain itself, then covers what AI may and may not do in it, the metadata that holds it together, readiness beyond functional tests, quality reporting, failed evidence, intermittent environments, readiness scores and guardrails for AI test automation.
The running example
Kudu Pharmacies is fictional, and its systems and figures are illustrative. Kudu is a South African pharmacy chain with about 180 stores and an app, Kudu Repeat, that lets patients request repeat prescriptions for collection or delivery. A nine-person product team ships a release every two weeks. The team uses an AI coding agent for implementation and an AI test generator that drafts test cases from acceptance criteria.
One requirement matters more than most. Requirement RX-214 says that a repeat request for a medicine needing pharmacist review must be held until a pharmacist approves it, and that no confirmation message may reach the patient before that approval. Get this wrong and a patient is told their medicine is ready when nobody has checked it. That is a clinical risk, a reputational risk and, depending on the medicine, a regulatory one.
In release 4.12 the team changed how held requests are queued. The test report for 4.12 showed 1,402 automated tests passing. The release manager asked a better question than "did they pass?". She asked: "Which of those tests prove RX-214 still holds?" Nobody could answer quickly. The tests existed. Their connection to RX-214 did not.
What orphan evidence looks like
Orphan evidence is not bad testing. Most of Kudu's tests were well written. The problem is that each one stood alone. A test called test_hold_queue passed, but it carried no reference to RX-214, to the acceptance criterion it was meant to prove, or to the build it ran against. To answer the release manager, someone would have had to read the test code, guess its intent and hope the guess matched the requirement.
Orphan evidence shows up in four recognisable ways:
- Tests named after code, not outcomes.
test_stock_apidescribes what it touches, not what it protects. - Coverage measured by lines, not by requirements. Seventy-four percent line coverage says nothing about whether the riskiest behaviour is tested.
- Results that cannot be tied to a build. A pass from Tuesday's build is quoted in Thursday's release meeting.
- Decisions with no record of what they relied on. The release went out, but nobody can show which evidence the approver saw.
AI test generation makes all four worse if nothing else changes. A generator asked to "write tests for the hold queue" will happily produce dozens of plausible tests. Without a requirement and criterion to anchor them, they multiply the count without improving the answer.
The evidence chain
The alternative is a chain with seven links. Each link records something the next one depends on.
Intent says why the work exists and who owns the risk. For RX-214, the intent is patient safety and the risk owner is a senior pharmacist acting as clinical owner for the app.
Requirement gives the intent an identifier, a statement and a risk class. RX-214 is classed high.
Acceptance criteria turn the requirement into testable conditions. RX-214 has three: a request for a review-required medicine enters a held state (AC-1); no confirmation is sent while it is held (AC-2); a pharmacist's approval releases exactly one confirmation (AC-3).
Test case cites the criterion it proves, the data it uses and the result it expects. TC-881 proves AC-3.
Execution records the build, environment, time and result. Run 4.12-0317 executed TC-881 against build b-5521 in the staging environment.
Evidence keeps the logs, traces and artefacts from that run, with a hash so the record cannot be quietly replaced.
Decision records the outcome, GO, NO-GO or conditional, with the name of the person who made it and the evidence they relied on.
Two of these links are where most organisations already have something: requirements usually exist in a backlog, and test results usually exist in a pipeline. The missing parts are the joins between them. Our earlier article on requirements, assumptions and traceability makes the same point for proposals: a requirement nobody can trace is a requirement nobody can prove. The release version of that problem is more expensive, because it ships.
Why the chain matters more when AI writes the tests
If an agent writes the code and another agent writes the tests, a green suite proves less than it used to. The tests may share the implementation's assumptions. They may test what the code does rather than what the requirement asked for. They may be numerous and shallow. That is one of the central risks of agent-written software, and the reason our Coding Agent Harness series treats checks as part of the environment around the agent: a check written by the same process that wrote the code is not an independent check.
The evidence chain gives AI-generated tests a job they can be held to. When the test generator is given AC-3 as its input and must cite AC-3 in every test it produces, three useful things happen:
- Coverage becomes countable by criterion. The team can see that AC-1 and AC-3 have tests, and AC-2 has none, before release day.
- Review becomes focused. A reviewer reads the five tests that claim to prove AC-2, rather than skimming four thousand tests that claim nothing in particular.
- Gaps become visible to the machine. A pipeline can refuse to mark a high-risk requirement as evidenced when any of its criteria has zero passing tests on the release build.
That third point is where the chain stops being documentation and becomes a control. It is the same move spec-driven development makes on the way in: the specification is no longer a description of intent, it is something the system checks against.
What Kudu changed
Kudu did not buy a new tool. It made five changes to the tools it already had, over two release cycles:
| Change | Where | Effort (illustrative) |
|---|---|---|
| Every requirement gets an ID and a risk class | Backlog | A morning's backlog cleanup |
| Acceptance criteria get IDs | Requirement template | Template change |
| Every test cites at least one criterion ID | Test naming and tags | Lint rule in the pipeline |
| Every run records build ID and environment | Test runner configuration | Configuration change |
| The release record lists high-risk criteria and their evidence | Release checklist | A generated table |
The lint rule did the most work. A test that cited no criterion still ran, but it was flagged as unanchored in the report. Within two releases, the team had retired about three hundred tests that nobody could attach to a requirement and that duplicated other coverage. The suite got smaller and the release decision got better informed. Those numbers are illustrative, but the direction is typical: anchoring tests tends to expose duplication.
The release manager's question about RX-214 now takes one query to answer. The release record for 4.12 shows three criteria, the tests that cite each one, their results on build b-5521 and the name of the person who signed the decision.
The boundaries of this approach
Traceability is not free and it is not universal. Three cautions keep it honest.
Not every test needs a business requirement. Low-level unit tests protect implementation details and refactoring safety. They can cite a component or a technical requirement, or simply be excluded from release evidence. The chain matters most for behaviour a customer, regulator or operator would notice.
A chain can be complete and wrong. If AC-2 is badly written, a test can prove it perfectly and still miss the real risk. The chain makes the reasoning inspectable; it does not make it correct. A human still has to read the high-risk criteria.
Traceability can become theatre. A team that tags every test with a random criterion to satisfy the lint rule has built a chain of false links. Sampling, a short weekly review of a handful of high-risk links, catches this quickly.
The decision you can make this month
Pick the five highest-risk requirements in your product. For each one, ask three questions: does it have an identifier, are its acceptance criteria written down, and can you list the tests that prove each criterion on the build you are about to release? If you cannot answer the third question in a few minutes, your release decisions are relying on orphan evidence.
Start there, with five requirements, not five hundred. The chain grows from the risky end.
Next: Part 2, AI can draft the test. It cannot own the release. All parts are on the series page.
How CloudNala can help
We help delivery teams build the evidence chain into the tools they already use: identifying the requirements that carry real risk, writing acceptance criteria that tests can cite, adding pipeline checks that flag unanchored tests and generating a release record a decision-maker can read. The work usually starts with a short review of one product's last few releases, so the chain is designed around the evidence you actually have rather than a template.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za