Every operations team has a story that starts the same way. The release passed every test. The suite was green, the demo went well, the approval was signed. Then, at 08:15 on the first Monday of the month, the system fell over, and the post-incident review found nothing wrong with the code at all. The certificate had expired. The new database index locked a table under real load. The alert went to a channel nobody watched any more.
None of those failures would have been caught by more functional tests. Functional tests answer one question: does the build do what its acceptance criteria say? That question matters, and the first three parts of this series were about answering it properly. But production readiness is a wider question, and a release gate that only checks functional evidence is checking the part least likely to cause an outage.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. Kudu's Kudu Repeat app serves about 180 stores, and the first business day of the month, when many chronic medication repeats fall due, is its busiest.
The six other questions
A practical readiness gate asks seven questions, each with its own evidence, dated for this release.
Functional. Does the build meet its acceptance criteria? Evidence: the chain from Part 1.
Security. Have the scans for this build been run and triaged, with no open critical findings? Evidence: scan results tied to the build digest, and a triage record for anything above low severity. "The scanner ran" is not evidence; "the findings were reviewed and these were accepted by this person" is.
Performance. Does it handle peak load plus a margin, on this build? Evidence: a load test result against a defined peak. For Kudu, peak is the first business day of the month, not an average Tuesday.
Resilience. What happens when a dependency fails? Evidence: a test where the medical aid claim service, the SMS gateway or the stock system is slow or unavailable, and the observed behaviour. The point is not that nothing breaks; it is that the team knows how it breaks.
Operability. Can the people on call run it? Evidence: alerts exist for the new behaviour, a runbook entry explains what to do when they fire, and a named on-call owner has read it.
Data. Will the data survive the change? Evidence: any migration rehearsed on a realistic copy, and a backup restored at least once recently. A backup that has never been restored is a hope, not a control.
Rollback. Can the release be reversed? Evidence: the rollback procedure exercised on this release, including any data migration that must be reversed or made backward-compatible.
What a green suite hides
At Kudu, a review of two years of release history (illustrative) found six production problems that sat beside fully passing functional suites. Each maps to one of the six questions:
- Peak load was never exercised, so a queue backed up on the first of the month. (Performance)
- A certificate on the claim integration expired, and nothing tested its renewal. (Resilience)
- The rollback script was two releases out of date. (Rollback)
- Alerts for the hold queue routed to a team that had been reorganised. (Operability)
- A migration was tested on a small copy of the data and timed out on the real volume. (Data)
- The pharmacist approval queue had no monitoring, so held requests aged unnoticed. (Operability)
Not one of these was a coding error that a better unit test would have caught. All of them were evidence gaps in categories the release gate did not ask about.
Keeping it proportionate
The obvious objection is that seven categories of evidence for every release will grind a fortnightly cadence to a halt. It would, if every release needed fresh evidence in every category. It does not.
Evidence has a shelf life, and the shelf life depends on what changed. A release that changes only copy on a screen does not need a new load test; last month's is still valid. A release that adds a database index needs a fresh performance and data rehearsal. The gate should ask "is there current evidence for this category, given what changed?", not "did we redo everything?".
A simple rule set does most of the work:
| If the release changes... | Then refresh evidence for... |
|---|---|
| Business logic on a high-risk requirement | Functional, rollback |
| A database schema or migration | Data, performance, rollback |
| An integration or dependency version | Resilience, security |
| Infrastructure, networking or certificates | Resilience, operability, security |
| Alerting, logging or on-call ownership | Operability |
| Nothing above (copy, styling, low-risk config) | None beyond functional |
The work item's risk class and a few labels on the change drive this automatically. Most releases need two or three categories refreshed, not seven.
Who owns each category
The other source of friction is unclear ownership. Functional evidence belongs to the product team. The other six often belong partly to platform, security, database or operations teams who are not in the release meeting. When nobody owns a category, the evidence is assumed.
The fix is a short ownership table, agreed once: who produces the evidence for each category, who reviews it and how long it stays valid. At Kudu, security triage sits with the platform security lead, performance with the product team using a shared load-test harness, and operability with whoever is on call for the following fortnight. The release manager does not produce this evidence; they check that it exists and is current.
A worked example: release 4.12
Release 4.12 changed how held requests are queued. That touched business logic on a high-risk requirement and added a database index. Under the rule set, it meant refreshing functional, rollback, data and performance evidence. For the hold flow itself, security, resilience and operability evidence from the previous release were still current, because nothing in those areas had changed.
The data rehearsal is the one worth describing. The index was first tested on the standard staging copy, about a tenth of production volume, where it built in under a minute. On a masked copy at full volume it took eleven minutes and locked the queue table while it ran. Nothing in the functional tests would have revealed this. The team changed the migration to build the index without locking, reran the rehearsal and attached the timing to the release record. The figures are illustrative, but the pattern is common: data volume is where staging environments most often mislead.
That single finding justified the whole category. It also shows why the evidence must be dated for this release. A data rehearsal from 4.11 would have said nothing about an index that did not exist yet.
The link to production
Readiness evidence is a prediction. Production is where the prediction is tested. The categories here line up closely with what an operations team watches after go-live, which is the subject of Production begins after deployment and, for the networking layer, designing for the bad day. A team that gathers readiness evidence and then watches the same signals in production learns quickly which of its predictions were right.
For AI features specifically, add an eighth category: evaluation evidence on a representative dataset, covered in Evals that gate real work. A model change can pass every functional test and still give worse answers.
Next: Part 5, Quality intelligence is more than a test dashboard. All parts are on the series page.
How CloudNala can help
We help teams define a proportionate readiness gate: the evidence categories that matter for their system, what triggers a refresh, who owns each one and how the release record shows it. We also run readiness reviews before significant go-lives, where an independent view of the evidence is most useful.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za