AI features fail in ways that feel new. A summariser invents a promise to pay. A retrieval change pulls the wrong customer's notes into a reply. A model update shifts the tone of every message overnight. When it happens, organisations tend to swing between two unhelpful responses: treating the failure as a mysterious property of AI that nobody could have prevented, or finding the nearest person who touched the output and making it their fault.
Neither produces a better system. The first learns nothing because it assumes nothing could be learned. The second learns the wrong thing because it fixes a person, and the next person in the same seat will face the same conditions.
CloudNala's view is that an AI failure needs the same governance as any other failure involving personal information, plus two things AI makes harder: reconstructing what the system actually did, and identifying the control that should have caught it. A review that ends in a changed, tested control has done its job. A review that ends in a name has not.
This series is general information about running privacy controls in production systems, not legal advice. It summarises the Protection of Personal Information Act for a technical and business audience; read the primary text and take advice from your information officer and a qualified legal practitioner before relying on any interpretation.
This is the final part of POPIA in Production. It draws on every earlier part: the control register from Part 1, the purpose attributes from Part 2, the retention of traces from Part 3, the person index from Part 4 and the classification path from Part 5.
Amanzi Credit is fictional, and its incident, systems and figures are illustrative. It is a Gqeberha lender whose AI assistant summarises every call-centre conversation into a CRM note that agents use for follow-up.
What happened
A developer improved the call-note summariser by giving it "recent context": before summarising a call, it retrieved the five most similar recent notes from the AI search index so it could recognise ongoing issues. The change passed code review and the summariser's evaluation set. The retrieval query, however, filtered by similarity and date, but not by customer.
For three days, some summaries included fragments of other customers' notes: a name, an outstanding balance, a mention of a payment arrangement. Agents saw them in the CRM. In 11 cases, agents pasted the summary into a follow-up email to the customer, as the process encouraged them to do. A customer who received another person's balance complained on day three.
Contain first
Amanzi's summariser had a feature flag, which is the single most useful control this story contains. Within 20 minutes of the complaint reaching the platform owner, summaries were switched off and agents were told to write notes by hand. The traces showed the exact time the retrieval change was deployed, so the affected window was known before anyone had finished their first meeting.
If your AI features do not have a kill switch that a named person can use without a deployment, add one before anything else in this article.
Classify, and know when it is not a compromise
The Part 5 path applied directly. Personal information was involved. Eleven emails had disclosed one customer's information to another customer, which, in the information officer's assessment, gave reasonable grounds to believe it had been acquired by unauthorised persons. Amanzi's information officer, with legal advice, notified the Regulator through its eServices portal and wrote to the affected customers, then updated the notification as the scope became clearer.
Summaries shown only inside the CRM raised a harder question: agents are authorised to see customer records, but not records of customers they are not serving. Amanzi took the conservative view, included those customers in its scoping, and recorded its reasoning. That decision belongs to the information officer and legal advisers, which is why the classification page exists.
Not every AI failure is a security compromise. It helps to know which condition a failure touches, because the response differs.
| AI failure | What it most likely engages | Response |
|---|---|---|
| Another customer's details appear in output that leaves the organisation | Security safeguards, section 22 notification | Classification path, notification |
| Summary records something the customer never said | Information quality (section 16), correction rights (section 24) | Correct the record, inform recipients, fix the model's instructions or evaluation |
| AI output feeds an adverse decision without human review | Automated decision-making (section 71) | Restore human decision, check representations route |
| Output reused for a purpose never assessed | Further processing (section 15) | Stop the use, run the gate from Part 2 |
| Output is offensive or wrong but contains no personal information | Not a POPIA matter in itself | Product quality process |
Reconstruct what the system did
To scope the incident, Amanzi needed to answer four questions: which calls were summarised during the window, which summaries contained foreign fragments, whose information those fragments came from, and where the affected summaries went next.
It could answer them because three things had been kept: the trace for each summarisation (inputs retrieved, prompt version, model version, output), the deployment history of the summariser, and the CRM's record of which notes were copied into emails. The person index from Part 4 then mapped every affected customer, in both directions.
There is a real tension here with Part 3. Traces contain personal information and should not be kept indefinitely. Amanzi's answer is a tiered rule: full traces, including content, are kept for 30 days in a restricted store; metadata without content (identifiers, versions, timings, retrieved record identifiers) is kept for 12 months. The incident was found on day three, well inside the content window. An incident found on day 45 could still have been scoped from metadata, because the retrieved record identifiers were logged. Decide what you will need to reconstruct before an incident, and keep exactly that.
Find the control that failed, not the person
The first draft of the review, written under pressure, had four recommendations. Every one of them was about people.
None of them survived the second draft. The agents had followed the process they were given. The developer's change was reviewed and passed the evaluation set. The failure was in the system around the people: the evaluation set had no case testing whether another customer's information could appear in a summary, the retrieval layer did not enforce customer scoping, and nothing checked the output before it reached an agent.
The corrective actions were therefore controls:
- retrieval enforces customer scoping in the query layer, so no prompt or code change can omit it;
- each summary cites the identifiers of its source records, so a foreign source is visible;
- an output check blocks any summary containing identifiers that do not belong to the current customer;
- the evaluation set gained adversarial cases with similar customers, shared surnames and recent notes, and the release gate requires them to pass;
- a weekly sample of summaries is reviewed by a person, with the error rate reported.
Each became a row in the control register, with an owner, the evidence it produces and a test. The Release on Evidence series covers the release-gate side of this in more depth.
Six steps, each leaving evidence
Amanzi now uses this sequence for every AI failure involving personal information. The right-hand column matters as much as the left. Months later, when an auditor, the Regulator or the board asks what happened and what changed, the answer is a set of records rather than a set of recollections.
What the board should see
The board did not need the technical detail. It needed one page with six items: what happened, in two sentences; how many people were affected and how they were told; the status of the notification to the Regulator; the control that failed, named as a control; the controls now in place, with the evidence that they work; and the decision being asked of it, which in this case was whether to restore the summariser.
That last item is the one most reviews forget. A failure review that does not end in a decision leaves the feature in limbo, usually switched back on quietly a few weeks later without anyone having agreed the conditions.
Re-enable on evidence
Amanzi restored the summariser 19 days later, on evidence agreed in advance: all adversarial evaluation cases passing, the output check active and tested with deliberately planted foreign identifiers, two weeks of shadow running with zero blocked outputs reaching agents, and the sampling review in place. The decision and its evidence were recorded in the register and reported to the board.
What this series adds up to
Across six parts, the argument has been the same. POPIA is met in production by running controls that produce evidence, not by documents that describe intentions. Purpose and lawful basis are attributes the platform can check. Retention is a pipeline. Access requests are a system capability. Incident classification is decided before the incident. And when an AI feature fails, governance is the discipline of finding the missing control and proving that it is now in place.
AI does not change any of those principles. It multiplies the places where personal information is created, copied and transformed, which makes each of them harder to do by hand. That is the case for building them into the systems, now, before the next feature ships.
Start this week
- Confirm every AI feature that touches personal information has a kill switch that a named person can use without a deployment.
- Decide what you would need to reconstruct an AI incident and check that your traces keep exactly that, for a defined period.
- Add one adversarial privacy case to each AI feature's evaluation set: can this feature ever show one person's information to another?
This is the last part of the series. Start again at Part 1, A privacy policy is not a privacy control, or see all parts on the series page.
How CloudNala can help
CloudNala helps organisations prepare for, and learn from, AI failures that involve personal information. We review your AI features for kill switches, trace retention and evaluation coverage, help you run a blameless post-incident review that ends in tested controls, and prepare the evidence and board summary that show what changed. We work alongside your information officer and legal advisers, who remain responsible for legal determinations and notifications.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a POPIA Controls Review or write to us at consult@cloudnala.co.za