Ask most teams running an AI-assisted decision process how they would explain one outcome from six months ago, and the answer involves a lot of searching. The application is in the case management system. The documents are in a storage bucket. The model call is in an observability tool with thirty days of retention. The rule that applied is in a code repository, at some commit. The reviewer's approval is a status change with a timestamp and a username. Putting those together is possible on a good day, for one case, by the engineer who built it.
That is not an explanation. It is an archaeology project. And it fails exactly when it is needed most: in an appeal, a complaint to a regulator, an audit or a court, months later, after the engineer has moved on and the logs have rolled over.
Amatola Bursary Agency, the fictional provincial agency in this series, decided early that every recommendation would produce a case file: one record, assembled as the case is processed, that holds everything needed to reconstruct the outcome. This part sets out what goes in it and why.
What the case file holds
Application inputs as received. The form data exactly as submitted, before any cleaning or correction. If a field was later corrected, the correction is recorded separately with who made it and why.
Evidence documents. The documents the applicant supplied, stored with a hash so that anyone can later confirm the file has not changed. The case file references them rather than copying them, which matters for retention, as discussed below.
Rule versions applied. Each rule the engine evaluated, by identifier and version, with the date used to select it. This is the link back to the rules register from Part 2.
Sources retrieved. If the assistant retrieved policy text, guidance or reference data, the case file lists what was retrieved and which version of each source. A model's explanation can then be checked against what it actually saw.
Model, prompt and tool versions. Which model, which prompt or instruction version and which tool versions were used. Without this, nobody can say whether a later change in behaviour came from the case or the system.
Tool calls and results. Every call the assistant or engine made to another system, such as a population register lookup or an academic results check, with the request and the response. These are often the evidence the decision actually rests on.
Recommendation and draft reasons. The outcome the system proposed, the rule trace from the engine and the draft reasons the assistant wrote.
Official decision and final reasons. What the responsible official decided, whether it matched the recommendation, any discretion exercised, the final reasons and who signed them.
Only the last item is the decision. Everything else is the evidence for it.
Recommendation and decision are separate records
The most important structural choice is to keep the system's recommendation and the official's decision as separate items. They often match. When they do not, the difference is the most informative thing in the file.
If an official overturns a recommendation, the case file records the reason. Patterns in those reasons tell the agency where the rules, the extraction or the model are wrong. If an official never overturns a recommendation over thousands of cases, that is also a signal: either the system is very good, or review has become a rubber stamp. The human in the loop must show their work covers the second risk in detail. The case file is what makes either pattern visible.
Merging recommendation and decision into one status field ("approved by system, reviewed by assessor") destroys that information. It also blurs accountability: it becomes unclear whether the official decided or merely did not object.
Assembled during the case, not reconstructed after it
A case file assembled after the fact from logs is a reconstruction, and reconstructions have gaps. Amatola's design writes to the case file at each step, as the step happens. The engine appends its rule trace when it runs. The assistant's calls are recorded by the orchestration layer, not by the model. The official's decision is written by the case management system when it is signed.
This has a useful side effect: the case file becomes the single place the rest of the process reads from. The reasons letter is generated from it. The appeal panel opens it. An auditor samples it. Nobody needs access to the observability tool or the code repository to understand a case, and those tools can keep their own shorter retention without destroying evidence.
The general point, that a trace of what a system did is not the same as evidence that it did the right thing, is made in A trace is not an evaluation. The case file is the decision-level counterpart: it is shaped around the question "can we justify this outcome?" rather than "what did the software do?".
Reasons that trace to rules
The reasons an applicant receives should be built from the case file, not written freshly. At Amatola, the reasons letter is assembled in three layers:
- The finding: which condition was not met, citing the rule and its source clause.
- The facts: the specific facts from the case that led to that finding, with the evidence they came from.
- The explanation: a plain-language paragraph, drafted by the assistant from the first two layers and reviewed by the official.
The first two layers come straight from the engine's trace and the confirmed inputs. The model only writes the third, and it writes from structured inputs rather than from the raw documents. That makes it much harder for an explanation to cite a rule that did not apply or a fact that was not in the case.
For public bodies in South Africa, adequate reasons are not optional. The Promotion of Administrative Justice Act gives a person whose rights are materially and adversely affected by administrative action the right to request written reasons, and the administrator must provide adequate reasons within 90 days of the request, as the Department of Justice's PAJA guidance explains. A case file is the practical way to meet that obligation for every case rather than only for the ones that ask. Human-in-the-loop is not administrative justice looks at the wider administrative-law picture. This is general information, not legal advice.
Retention without hoarding
A complete case file contains personal information, sometimes a lot of it. That creates a tension with POPIA's requirement, in section 14, not to keep personal information longer than necessary. The answer is not to keep less evidence; it is to design retention deliberately.
Amatola's approach, illustrative and subject to its own legal advice:
- The case file keeps references and hashes to evidence documents, so documents can follow their own retention schedule while the file still proves what was considered.
- Model inputs and outputs are stored in the case file only to the extent needed to reconstruct the recommendation; full raw prompts are kept for a shorter period in the observability tool.
- The retention period for the case file is tied to the period during which the decision can be challenged, plus a defined buffer, and is recorded with the reason.
- Deletion is itself logged, so the agency can show a case file was removed on schedule rather than lost.
The POPIA in Production series covers retention as a deletion pipeline in more depth.
A test you can run this week
Pick one decision your system made three months ago. Without asking the engineer who built it, try to answer: what inputs did it see, which rule version applied, what did the model retrieve and produce, who decided, and why? Time how long it takes and list what you could not find. That list is your case file specification.
Next: Part 5, An appeal is a reassessment, not a second opinion. All parts are on the series page.
How CloudNala can help
CloudNala helps teams design the case file for an AI-assisted decision process: what it holds, which component writes each part, how reasons are generated from it, and how it meets retention obligations. We run the three-month reconstruction test with you, map the gaps and design the changes to your orchestration and case management so that every recommendation can be explained from one record.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Decision Review or write to us at consult@cloudnala.co.za