AI & Data31 October 20268 min read

POPIA in Production — Part 3 of 6

Retention is a deletion pipeline, not a schedule

Most retention schedules are spreadsheets that nothing executes. Part 3 of POPIA in Production treats retention as a pipeline: a rule catalogue, triggers from systems of record, an executor for every store including vector indexes and traces, and a deletion record that proves it ran.

#POPIA#Privacy Engineering#Data Governance#Risk Management#AI Governance

Almost every organisation has a retention schedule. It is usually a spreadsheet, owned by records management or legal, with rows such as "customer records: five years after closure" and "call recordings: twelve months". It was carefully researched. It is periodically updated. And in most organisations, nothing reads it.

The systems that hold the data were configured by different people at different times, often with retention set to "keep everything" because storage is cheap and deleting things feels risky. The schedule describes what should happen. The infrastructure does something else. Nobody notices until an access request, a breach or a dispute reveals records that should not exist.

CloudNala's view is that a retention rule is only real when a pipeline deletes, de-identifies or restricts every copy of the information on time, and writes a record showing that it did. The schedule is the specification. The pipeline is the control. The deletion record is the evidence.

This series is general information about running privacy controls in production systems, not legal advice. It summarises the Protection of Personal Information Act for a technical and business audience; read the primary text and take advice from your information officer and a qualified legal practitioner before relying on any interpretation.

This is Part 3 of POPIA in Production. Part 2 put a retention rule among the attributes every store of personal information carries. This part is about what executes that rule.

Amanzi Credit is fictional, and its systems and figures are illustrative. It is a Gqeberha lender with about 180,000 active customers and, it discovered during this work, records for about 410,000 people who had closed accounts or been declined over the past decade, many of them well past any period its schedule allowed.

What section 14 requires

Section 14(1) says records of personal information "must not be retained any longer than is necessary for achieving the purpose for which the information was collected or subsequently processed", unless retention is required or authorised by law, reasonably required for lawful purposes related to the responsible party's functions or activities, required by a contract between the parties, or consented to by the data subject.

Section 14(4) then requires the responsible party to "destroy or delete a record of personal information or de-identify it as soon as reasonably practicable" after it is no longer authorised to retain it. Section 14(5) adds that destruction or deletion must be done "in a manner that prevents its reconstruction in an intelligible form". Section 14(6) requires processing to be restricted, rather than the record deleted, in specific situations: while the data subject contests its accuracy, where the information is no longer needed but must be kept for proof, where processing is unlawful and the data subject asks for restriction instead of deletion, and where the data subject asks to transmit it to another automated system.

Two words in those provisions matter for engineers. "Record" is defined broadly in section 1 and includes information stored by computer equipment "and any material subsequently derived" from it. "As soon as reasonably practicable" is a timing obligation. A schedule nobody executes satisfies neither.

This series does not prescribe retention periods. Amanzi's legal team sets them from the laws that apply to a credit provider, including record-keeping duties in financial and tax legislation, and records the legal reason against each rule. One provision matters for a lender in particular: section 14(3) requires a record used to make a decision about a person to be kept for any period prescribed by law or a code of conduct or, if none is prescribed, long enough to give the person a reasonable opportunity to request access to it. A pipeline that deletes declined applications too quickly can fail that duty, so Amanzi's rule catalogue flags decision records separately. The engineering problem starts once the period is known.

Where the copies actually are

Amanzi began by following a single closed account through its estate. The customer had closed their store card in 2019. Under Amanzi's own schedule, their personal information should have been deleted or de-identified by early 2025. It was found in nine places: the core loan system, the CRM, the warehouse (twice, in a raw table and a reporting table), call recordings, the vector index used by the AI search tool, application trace logs, nightly backups, and the systems of two operators.

There were also three spreadsheets on a shared drive, exported for a collections project in 2020. Those were the hardest to find and the hardest to delete.

Retention is a pipeline that reaches every copyTRIGGERAccount closed+ rule periodCRM and core loan systemDeleteData warehouse tablesDelete or de-identifyCall recordingsDeleteVector index of call notesRemove and re-indexPrompt and trace logsExpire by time limitBackupsExpire with rotationCopies at 3 operatorsInstruct and confirmDeletion record: rule, run date, counts per store,operator confirmations, exceptions
One retention trigger at Amanzi fans out to the seven kinds of store where the same customer's information lives. Each store has its own mechanism, and the pipeline writes one deletion record per run. Backups are not deleted record by record; they expire on a rotation the record states, so the gap is visible rather than hidden.

The diagram shows the pipeline Amanzi built. It has four parts.

A rule catalogue. Each rule has an identifier, a trigger event, a period, the legal reason for retention under section 14(1), and the action at the end: delete, de-identify or restrict. Rule R-07, for example, is "account closure plus the statutory record-keeping period, then delete or de-identify".

Triggers from systems of record. The clock starts on an event recorded by an authoritative system: account closed in the core loan system, application declined in the origination system, call ended in the telephony platform. A retention rule that depends on someone remembering to start it is not a rule.

An executor per store. Each store has its own mechanism: a delete job in the CRM, partition drops in the warehouse, a platform setting for recordings, a metadata-filtered delete for the vector index, a time-to-live for logs. The executors are boring on purpose. They are small, tested and owned.

A legal-hold override. When a dispute, investigation or litigation requires records to be kept, a hold is recorded against the person or account, and every executor checks it before acting. Holds have owners and expiry dates, because a hold that never ends quietly becomes indefinite retention.

The awkward stores

The core systems were the easy part. Five other kinds of store needed specific decisions.

Backups. Deleting individual records from immutable backups is usually impractical. Amanzi's approach is to keep backups on a defined rotation, state that rotation in the deletion record, and maintain a "deletion replay" list: if a backup is ever restored, the pipeline re-applies every deletion made since that backup was taken before the restored system is used.

Logs and traces. Application logs and AI trace logs had captured full prompts, including call transcripts. The best control here is upstream: minimise what is logged, redact identifiers at write time where possible, and keep a short time-to-live on anything that still contains personal information. Debugging convenience does not create a lawful purpose for keeping transcripts in a log store for years.

Vector indexes. Amanzi's AI search tool embedded chunks of call notes, each with a customer number in its metadata. Deletion means removing every chunk for that customer by metadata filter and confirming with a follow-up query that none remain. Amanzi treats embeddings of call notes as personal information unless it can show otherwise, because they are derived from personal information and linked to an identifier.

Model providers. If prompts are sent to an external model service, that provider's retention settings for inputs and outputs are part of your retention story. Check what the contract and the service configuration say, and record it.

Operators. Section 21 makes security measures at operators the responsible party's concern through contract. Amanzi extended the same thinking to retention: each operator receives deletion instructions from the pipeline and returns a confirmation, and missing confirmations appear in the deletion record as exceptions.

The spreadsheets were dealt with the old-fashioned way: found by searching the shared drive for identifier patterns, deleted, and followed by a rule that exports of personal information go to a managed location with its own expiry.

Delete, de-identify or restrict

Section 14(4) offers a choice between destroying or deleting the record and de-identifying it, and section 14(6) requires restriction in some cases. The three end states are different, and teams often blur them.

Delete, de-identify or restrict: three different end statesACTIONWHAT REMAINSSTILL PERSONAL INFO?TYPICAL USEDeleteNothing that can bereconstructedNoPurpose ended andno legal hold appliesDe-identifyRecords with identifiersremoved per s1 testNo, unless it canbe re-identifiedStatistics andtrend reportingRestrictThe full record, withuse blockedYesProof, disputes,contested accuracy
Section 14(4) allows a record to be destroyed, deleted or de-identified once retention is no longer authorised, and section 14(6) requires restriction in some cases instead. De-identification only counts if the person cannot be re-identified by a reasonably foreseeable method, which is a test to run, not a label to apply.

De-identification is a test, not a label. Section 1 defines it as deleting any information that identifies the data subject, can be used or manipulated by a reasonably foreseeable method to identify them, or can be linked by a reasonably foreseeable method to other information that identifies them. Removing names and leaving a hashed ID number usually fails that test. South African ID numbers have a known structure that begins with a date of birth, so an unsalted hash of one can, in our assessment, be reversed by enumerating plausible numbers. Amanzi's reporting tables replaced hashed IDs with random surrogate keys and dropped the mapping at the end of the retention period.

Restriction keeps the record but blocks its use. It is the right state while a customer disputes the accuracy of a record or where Amanzi must keep it as proof, and it needs a technical mechanism: a flag every consuming system respects, not a note in a ticket.

Deletion has to prevent reconstruction. For most cloud stores that means deleting through the service's own mechanisms and relying on its documented handling of the underlying storage, recorded in the register. For local media and paper, it means destruction methods suited to the medium.

Proving it ran

Every run of the pipeline writes a deletion record: the rules applied, the run date, counts per store, operator confirmations received, holds that blocked deletion, and exceptions. Amanzi tracks one headline measure per store, the age of the oldest record past its retention date, and reports it monthly to the information officer. When that number is zero, the rule is working. When it grows, something has broken, and the control register row from Part 1 shows who owns the fix.

Quarterly, someone who does not own the pipeline picks a sample of closed accounts past their retention date and searches for them in every store, including the vector index. Finding one is a finding, not an embarrassment, and it is handled the way the Technology Health Check series handles any other finding.

For the AI-specific stores, the Memory Is an Architecture Decision series covers retention for agent memory in more depth.

Start this week

  1. Follow one closed customer through your estate and list every place their information still exists, including logs, indexes, backups, operators and spreadsheets.
  2. Pick your largest store and measure the oldest record past its retention date. That single number tells you whether the schedule is being executed.
  3. Write the deletion record format before the pipeline. Deciding what you will need to prove clarifies what each executor must do.

Next: Part 4, An access request is a test of your whole system. All parts are on the series page.

How CloudNala can help

CloudNala designs retention pipelines that reach every copy. We trace sample records across your systems, operators and AI stores, turn your retention schedule into a rule catalogue with triggers and executors, decide with your teams how backups, logs and vector indexes are handled, and set up the deletion record and tests that show the pipeline is working. Retention periods and legal reasons remain decisions for your legal advisers.


Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book a POPIA Controls Review or write to us at consult@cloudnala.co.za