Test dashboards have never looked better. Live charts of executions, pass rates trending upward, coverage percentages in large friendly numbers, and now AI-written summaries that explain the week's results in a paragraph. Vendors increasingly call this "quality intelligence".
Much of it is activity reporting with good design. It tells you how much testing happened. It rarely tells you how far the release decision can rely on that testing, or where the evidence is thin. Quality intelligence, if the term is to mean anything, is the second thing: a view of evidence quality, risk coverage, confidence and unresolved exceptions, organised around the decisions people actually have to make.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. The Kudu Repeat team's existing dashboard showed 1,402 tests executed, a 98.6% pass rate and 74% automation coverage for release 4.12. All three numbers were accurate. None of them told the release manager whether to ship.
Activity measures and evidence measures
The left column in the diagram is what most dashboards show. Each has a counterpart on the right that a decision-maker can use.
Tests executed becomes high-risk criteria evidenced. 1,402 tests is a measure of effort. "31 of 34 high-risk acceptance criteria have passing evidence on this build" is a measure of what is known. The three missing criteria are a short, specific list a person can act on. At Kudu, all three were evidenced before the release meeting, because an unevidenced high-risk criterion is a hard gate (Part 8) that no score or conditional can outvote.
Pass rate becomes failures explained. A 98.6% pass rate can hide twenty failures that were rerun until green without anyone recording why. The evidence measure counts failures that were classified, with a cause, against those that simply disappeared on rerun. Part 6 explains why the second group is so dangerous.
Automation coverage becomes coverage of the riskiest journeys. Seventy-four percent of what? Line coverage and test-case counts treat a login screen and a medicine hold the same. Coverage weighted by risk does not.
Defects found becomes open exceptions and their expiry. Finding defects is good. What a release owner needs to know is which known defects are being carried into production, who accepted them and when they expire.
Execution time becomes evidence age. A fast suite is pleasant. The more useful measure is how old each piece of evidence is at decision time, and whether it was produced on the build being released.
Organise by risk, not by count
The single most useful view in quality reporting is a grid: the product's journeys down one side, ranked by risk, and the types of evidence across the top.
Kudu's grid for release 4.12 showed four gaps. Three were on high-risk journeys: no exploratory testing of the changed pharmacist hold, and no performance or security evidence for the medical aid claim journey on this build. A coverage percentage would have averaged these away. The grid put them in front of the release manager, who asked for exploratory testing of the hold and for the claim journey performance and security runs before the release meeting. All three came back with passing evidence, so the meeting decided on a complete set rather than a conditional.
Building this grid does not need a new product. It needs the risk class on each requirement from Part 3, journeys mapped to requirements once, and test results tagged with evidence type. A short script can produce it from data most teams already hold.
Confidence is a statement, not a number
Some tools now produce a "confidence score" for a release. Part 8 deals with scores directly. Here the point is narrower: confidence in evidence comes from a handful of properties that can be stated plainly.
- Independence. Was the test written by the same agent or person who wrote the code? Independent evidence carries more weight.
- Relevance. Does the test exercise the behaviour the requirement describes, or a proxy for it?
- Currency. Was it run on this build, in an environment like production?
- Completeness. Does every acceptance criterion on a high-risk requirement have evidence, including the negative cases?
- Integrity. Is the record intact, with failures retained and nothing overwritten?
A quality report that says, for each high-risk requirement, which of these properties hold is more useful than any single confidence figure. It tells the reader what to trust and what to check.
Where AI summaries help and where they mislead
AI summaries of test results are useful for one audience and risky for another.
For engineers investigating failures, a summary that clusters failures by likely cause, links to traces and suggests where to look first saves real time. The engineer reads the evidence anyway; the summary is a map.
For approvers, the same summary can mislead. "Two minor failures, both environmental, no impact on release readiness" is a conclusion, not evidence. If the approver reads only the summary, the agent has effectively made the classification and the decision. The safer pattern is that summaries for approvers are structured, listing each high-risk criterion, its evidence and any open exceptions, rather than prose that interprets them. Our Agent Evidence Layer series makes the related point that a trace of what happened is not the same as an evaluation of whether it was right.
Measures that improve behaviour, and measures that distort it
Any measure that is reported will be managed, and some measures are easier to manage than to improve. Pass rate is the clearest example. A team under pressure to raise its pass rate can retry flaky tests until they pass, delete tests that fail often, or loosen assertions. Every one of those raises the number and weakens the evidence.
Evidence measures are harder to game in the same way, because they point at specific gaps rather than an average. "High-risk criteria evidenced: 31 of 34" can only be improved by evidencing the three missing criteria, or by honestly reclassifying their risk, which leaves a record. "Failures explained" can only be improved by explaining failures. That is the property to look for in any quality measure: the cheapest way to improve the number should also be the useful thing to do.
Two measures from the activity column remain worth tracking internally, as long as they are not targets. Suite duration matters to engineers, because slow feedback changes behaviour. Flaky-test rate matters to the quality function, because a suite people do not trust is a suite people ignore. Neither belongs on a release owner's screen.
What to put in front of whom
Different people need different views, and one dashboard for everyone tends to serve nobody.
| Audience | What they need | Update rhythm |
|---|---|---|
| Engineers | Failures with traces, flaky-test patterns, slow suites | Continuous |
| Release owner | High-risk criteria and evidence, gaps grid, open exceptions | Per release |
| Product and clinical owners | Their requirements' evidence and accepted risks | Per release |
| Leadership | Escaped defects, exception ageing, evidence gaps trend | Monthly |
The leadership view deserves a note. The most informative quality measure for leadership is not anything a test dashboard shows. It is escaped defects: problems found in production that the release evidence should have caught. That number, and the categories it falls into, says more about the health of the quality process than any pass rate.
Next: Part 6, Why failed tests must keep their evidence. All parts are on the series page.
How CloudNala can help
We help teams turn the test data they already have into decision-ready quality views: a risk-weighted coverage grid, evidence measures for release owners and an escaped-defect view for leadership. The work starts from your existing tools and data, not a new platform.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za