Release meetings are short and evidence is long. So it is no surprise that teams, and increasingly their tools, compress everything into a number. "Release readiness: 87." Above some line it ships, below it does not. AI-assisted quality tools are particularly fond of these scores, because a model can weigh many signals and produce one figure that looks considered.
A score can be genuinely useful. It sorts releases, highlights the ones that need a closer look and trends over time. The danger is that the number replaces the reasoning. Once an approver sees 87, the inputs, the weights, what was left out and what someone overrode all disappear from view. A score without its explanation is not evidence. It is a conclusion with the working removed.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. Kudu's quality tooling introduced a readiness score for the Kudu Repeat app last year. Release 4.12 scored 87.
What sits behind the number
Kudu's score combines four inputs, each normalised to a hundred and weighted:
- Criteria evidenced (40%). The share of acceptance criteria with passing evidence on this build, weighted towards high-risk requirements.
- Open defects by severity (25%). Fewer and less severe open defects score higher.
- Non-functional evidence (20%). The readiness categories from Part 4 that apply to this change, with current evidence.
- Evidence freshness (15%). How recently the evidence was produced, relative to the build.
Two other things affect the number and are easy to lose. Exclusions: for 4.12, two suites were skipped because a test environment was down, and the score was calculated without them. Overrides: someone may have manually adjusted an input, for example marking a known defect as not blocking.
Every one of those elements is a judgement. The weights are a judgement about what matters. The exclusions are a judgement that missing evidence can be ignored. The overrides are judgements by named people. The number hides all of them.
Three ways a score misleads
The average hides the outlier. A release can score 92 overall with one high-risk criterion completely unevidenced, because the other thirty-three criteria pull the average up. The one gap might be the thing that hurts a patient.
Exclusions inflate the result. If the suites that were skipped were the ones most likely to fail, the score rises precisely because the evidence is weaker. Kudu's 87 would have been lower with the two skipped suites included. Nobody knows by how much.
Weights drift without review. Weights are set once, often by whoever built the tool. Months later, nobody remembers why freshness is worth 15% and security is folded into non-functional. The score keeps producing numbers with an authority it never earned.
AI-generated scores add a fourth: the weights may not be visible at all. A model that reads many signals and outputs "87, ready to release" may be right, but if it cannot show which signals drove the number, the approver cannot check it. That puts the score in the same position as the AI summary in Part 5: useful as a pointer, unsafe as a decision.
Rules that keep a score honest
A readiness score can stay in use if it follows five rules.
- Every score is shown with its inputs and weights. The approver sees the four components, not just the total.
- Exclusions are listed beside the score. "Calculated without: claim journey performance suite (environment unavailable)." If exclusions exceed a threshold, the score is marked incomplete.
- Overrides are recorded with who, why and until when. An override without a reason and an expiry is not allowed.
- Weights are owned and reviewed. A named person owns the scoring model and reviews it periodically against escaped defects. If high-scoring releases keep causing incidents, the weights are wrong.
- Hard gates sit outside the score. Certain conditions mean NO-GO whatever the number says.
Bands and hard gates
Kudu uses the score to sort releases into three bands. Below 70 is NO-GO. Between 70 and 85, the release needs a named approver to review the inputs and accept the gaps as a conditional GO. Above 85, the release can go ahead, provided no hard gate has failed.
The hard gates are the important part. For Kudu they are:
- any high-risk acceptance criterion that is unmet or has no evidence;
- any open critical security finding on this build;
- any data migration that has not been rehearsed on realistic data.
A release that scores 95 with an unrehearsed migration is NO-GO. The score cannot outvote the gate, because the gate represents a risk the organisation has decided it will not trade off against anything else.
This structure matches how the release decision was framed in Part 2: GO, conditional GO and NO-GO, with conditional outcomes requiring an owner, a mitigation and an expiry. The score helps sort releases into those outcomes faster. It does not make the decision.
Checking whether the score predicts anything
A score is a model, and a model can be tested. The test is to compare past scores with what happened after release. Take the last twenty or thirty releases, note each one's score and band, and list the escaped defects and incidents that followed each. If releases above 85 have as many escaped problems as releases between 70 and 85, the score is not separating good releases from risky ones, and the bands are giving false comfort.
Kudu ran this check after six months. The score separated releases reasonably well overall, but two of the three most serious incidents followed releases scoring above 90. Both had a single unevidenced high-risk criterion averaged away by strong results elsewhere. That finding is what led Kudu to add its first hard gate. The numbers are illustrative; the lesson is general. A score that is never checked against outcomes will keep its weights long after they stop being right.
Whoever owns the scoring model should repeat this comparison regularly and share the result with the teams that rely on the score.
Is a score worth having at all?
For a small team with a handful of high-risk requirements, probably not. A one-page release record listing the high-risk criteria, their evidence and the open exceptions is clearer than any number.
For a larger portfolio, with many teams releasing frequently, a score earns its place as a triage tool. It lets a platform or quality function see which releases across the organisation need attention and track whether evidence quality is improving over time. Used that way, the score is a management signal about the process, not a verdict on any single release.
The test of whether a score is being used well is simple. Ask an approver what drove the last score they relied on. If they can name the inputs, the exclusions and any overrides, the score is working. If they can only remember the number, it has replaced the reasoning.
Next: Part 9, AI test automation needs its own guardrails. All parts are on the series page. For the board-level version of this problem, where a single AI value number hides the evidence behind it, see The Boardroom AI Test.
How CloudNala can help
We review readiness scores and quality gates: whether the inputs and weights reflect real risk, how exclusions and overrides are recorded, which conditions should be hard gates and whether the score has predicted escaped defects. Where a score is not earning its place, we help replace it with a clearer release record.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za