A private company that releases a weak AI feature loses customers. A public body that releases a weak AI-assisted decision process can deny people licences, grants or permits they were entitled to, at volume, and may not find out until the appeals arrive months later. The bar for evidence is different, and so is the question the evidence has to answer.
The question is not "is the model good?" It is "does this service, with this model, these prompts, this rule version and these tools, reach the decisions the law requires, for the right reasons, across the full range of cases it will meet?" And it has to be answered again every time any of those parts changes.
That is what an evaluation harness is for. This part describes one for administrative AI, using the fictional Thukela Provincial Licensing Authority from Part 1, whose AI assistant prepares liquor-licence renewal assessments for officials to decide.
Why a demonstration proves almost nothing
Most public-sector AI procurements are judged on a demonstration: a handful of well-chosen cases processed live, with impressive speed and fluent reasons. A demonstration shows that the system can work. It says nothing about how often it fails, which kinds of case it fails on, or whether next month's model update will change its answers.
The evaluation literature that CloudNala has covered elsewhere, notably in Evals that gate real work, applies here with one important difference. In a commercial setting, "good" is often a matter of quality and preference. In administration, "good" is mostly defined by law. That makes the expected outcome for many cases knowable in advance, which makes rigorous evaluation more achievable, not less.
The harness, part by part
The scenario library is a set of realistic cases, each with its facts, documents and the correct outcome. It is the single most valuable asset in the programme.
The expected outcome and reasons for each case are written by people with authority to say what the law requires: experienced officials and the policy owner, with legal advice where needed. Each expected outcome cites the clause and rule version it rests on.
A rubric for the reasons text sets out what adequate reasons must contain: the rule applied, the material facts, how the facts meet or fail the rule, and what the applicant can do next. Reasons can be scored against it by trained reviewers and, once calibrated against them, by an AI judge. The Whoever Writes the Rubric series explains how to write and calibrate such a rubric.
Release thresholds say what the service must achieve before a change can go live: for example, zero wrong outcomes on exception and conflict cases, no new failures compared with the last release, and a minimum reasons score.
The runner sends every scenario through the real service, in a test environment, and records the outcome, the reasons, the rule version cited and the trace of what the AI did.
The comparison checks each result against the expected outcome and against the previous release, so that any case that has flipped is visible.
The release owner, a named official, reviews the report and accepts or rejects the change. The report is kept with the release record.
What goes in the scenario library
A library of routine cases will produce reassuring numbers and very little protection.
Thukela's library starts with about two hundred illustrative cases. Around a third are routine renewals. The rest are deliberately difficult: premises exactly at the distance threshold, applications lodged on the day a regulation changed, cases where a provincial rule and a municipal by-law point in different directions, applications with contradictory documents, and documents containing text designed to mislead the AI step. The most valuable group is historic appeals: real past cases, anonymised, where an official's decision was overturned. They show exactly where the organisation has erred before.
Two rules keep the library honest. It is owned by the policy unit, not the supplier, because whoever writes the test decides what "correct" means. And it grows: every new appeal outcome, every complaint that reveals a wrong decision and every encoding change adds cases.
Measure what administrative law cares about
The harness should report a small number of measures that map to real obligations.
| Measure | Why it matters |
|---|---|
| Wrong outcome rate, by scenario type | A wrong refusal and a wrong grant have different harms; report both |
| Unsupported reasons rate | Reasons that do not cite the rule or the material facts fail the applicant and the reviewer |
| Escalation accuracy | Did the system flag the cases it should not prepare alone, such as conflicts and open standards? |
| Version accuracy | Did it apply the rule version in force on the relevant date? |
| Flips since last release | Any case whose outcome changed needs an explanation |
| Consistency across similar cases | Applicants in the same position should be treated the same way |
Overall accuracy on its own is the least useful number. A service can be 97 percent accurate and still wrongly refuse every applicant in a minority situation the library under-represents.
Every change reruns the library
A public AI service changes more often than its owners expect. The supplier updates the model. Someone improves a prompt. The policy owner approves a new rule version. A tool the AI calls starts returning data in a slightly different format. Any of those can change outcomes.
The harness should run on every one of them, and the release owner should see which cases changed and why. Rule changes deserve special care: when a regulation is amended, the expected outcomes for affected scenarios must be updated by the policy owner first, so that the harness can confirm the service now applies the new rule, and only to cases from the effective date. Treating the library and rubric as versioned artefacts prevents a change in the test from being mistaken for a change in the service.
Fairness is a scenario property, not a slogan
Section 9 of the Constitution and the general duty of fair administrative action mean that consistency across groups is not optional. The harness can test it directly. Build paired scenarios that differ only in characteristics that should not matter, such as the applicant's language, the township or suburb of the premises, or the format of an uploaded document, and check that outcomes and reasons are the same.
Where the AI step extracts facts from documents, test it on the real variety of documents applicants submit: handwritten forms, scanned municipal letters, documents in more than one language. A model that reads typed English well and handwritten isiZulu poorly will produce unequal outcomes without anyone intending it.
Evidence that survives scrutiny
The harness produces more than a release decision. It produces evidence that the Auditor-General, an internal auditor, an ombud or a court might later ask for: what the service was tested on, what it got wrong, who accepted the risk and when. Keeping that evidence with each release record means the authority can answer the question "how did you know it was safe?" with documents rather than recollection.
The harness also gives the authority leverage over its supplier. A contract that requires the supplier's changes to pass the authority's scenario library, rather than the supplier's own tests, keeps the definition of "correct" with the public body. Part 7 returns to this in procurement terms, and the Machine-Readable Policy series separates testing the rule, the model and the whole decision.
Start this week
- Pull the last fifty decisions that were appealed or reviewed, and record the final outcome and reason for each.
- Write expected outcomes for twenty routine and twenty difficult cases, with clause references.
- Draft a one-page rubric for adequate reasons and test it on ten real decision letters.
- Agree who in the authority will own the library and who will act as release owner.
- Put a clause in the next AI-related procurement requiring changes to pass the authority's library.
Next: Part 4, Human-in-the-loop is not administrative justice. All parts are on the series page.
How CloudNala can help
CloudNala helps public bodies build the evaluation harness before they buy or build the AI. We work with your officials and policy owners to assemble a scenario library from real decisions and appeals, write expected outcomes and a reasons rubric, define release thresholds and set up the runner and reporting, so that every change to an AI-assisted service is tested against your definition of a correct decision, not a supplier's.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Start a Digital Discovery Sprint or write to us at consult@cloudnala.co.za