The question reached the security team at Fynbos Capital Bank, a fictional South African bank, on a Thursday afternoon. The case is illustrative. A confidential corporate-finance mandate, code-named in the bank's systems, was being discussed in the market before any announcement. The investigation team wanted to know whether anyone had used an AI tool with the mandate documents, and if so who, when and with which tool.
For the general assistant, the answer took an hour. The gateway recorded who had made each request, under which entitlement, and which internal documents the assistant had retrieved, each carrying its classification label. Two people on the deal team had used it to summarise a term sheet, as their entitlement allowed. Nothing suggested the documents had left the bank by that route.
For the coding agents, there was no answer. The sandbox recorded that sessions had happened. It did not record which files the agents had read, which commands they had run or which network destinations they had reached. If a coding agent had been involved, the bank could not show it either way. The leak turned out to have nothing to do with AI. The gap in the evidence was still the most important finding of the week.
Two failure modes
Organisations tend to fail at AI logging in one of two directions.
The first is to capture everything: every prompt, every response, every file, kept indefinitely. This feels safe. In practice it creates a large, sensitive store of customer information, employee information and confidential business material, copied out of the systems that were designed to protect it. Do your observability tools store the prompt? explains how these copies multiply across gateways, tracing tools and backups, and why logging is a data-handling decision.
The second is to capture nothing useful: licence counts and daily active users, which answer a procurement question and nothing else. Fynbos's coding-agent gap was a version of this. The record existed, but it described usage, not actions.
The way between them is to decide which questions the logs must answer, and capture the least that answers them.
Five questions, five layers
Fynbos identified five questions its AI logs must be able to answer, and built one layer for each.
Layer 1: entitlement events. Who was granted, changed or lost access to which capability, under which use case, approved by whom and when. Answers: was this person authorised? This layer lives in the identity governance system and follows the lifecycle in part 6.
Layer 2: session metadata. Who used which capability, when, from which device and environment, against which model, under which entitlement, with how much consumption. Answers: who used what? It contains no prompt or output content.
Layer 3: action records. What the capability did on the person's behalf: documents retrieved and their classification, connectors called, commands run, files read and written, network destinations reached, repository operations performed. Answers: what did it touch? This is the layer Fynbos was missing for coding agents, and for agents it is the most important layer.
Layer 4: content capture. The prompts and outputs themselves, where justified. Answers: what was said? This is the most sensitive layer and the one that should vary most by tier.
Layer 5: reliance records. When an output was relied on in a consequential decision, a link from the decision record to the evidence behind it. Answers: what did the organisation rely on? The prompt is part of the evidence sets out what that evidence pack should contain. Here the point is only that the log must be able to find it.
Log actions always, content selectively
Fynbos captures layers 1 to 3 for every capability and user, and layer 4 only where a stated purpose requires it, applying the per-field reasoning in Do your observability tools store the prompt? to staff use. When the mandate question arrived, what the bank needed was the list of documents retrieved and their labels, not the text of every summary.
Capture by capability and tier
The capabilities from part 1 differ in what they can do, so their logging differs too.
| Capability | Layers 1 to 3 | Layer 4 (content) | Layer 5 (reliance) |
|---|---|---|---|
| General assistant | Always; retrieved document IDs and labels in layer 3 | Not by default; enabled for tier 3 use cases | Where a tier 2 or 3 output is relied on |
| Sandboxed coding agent | Always; commands, file paths, network destinations and package installs in layer 3 | Not by default; the code itself is in version control | Through the pull request and review record |
| Analytics copilot | Always; queries and data products read in layer 3 | Queries kept; narrative output only for tier 3 | Where an output feeds a report or decision |
| Restricted change agent | Always, in full | Full session capture, access-restricted | Every change, through the pipeline record |
Two details matter. For coding agents, the version-control system already holds the most important content, the code, with authorship and review attached, so separate content capture adds little. For the restricted change agent, full capture is justified because the agent prepares changes to production and the set of users is small.
Logging staff is processing their personal information
Every one of these layers is a record about an identifiable employee. Under the Protection of Personal Information Act, that makes the logs personal information, and the ordinary conditions apply: a defined purpose, adequate security, and retention no longer than necessary. Section 14 requires that records not be kept "any longer than is necessary" for the purpose, subject to exceptions such as retention required or authorised by law, and that they be deleted, destroyed or de-identified once the authority to keep them ends.
Fynbos took three practical steps. It published a staff notice describing each log layer, what it contains and what it is used for. It set a retention period per layer, with the content layer the shortest. And it restricted access to content logs so that reading them requires a recorded reason and a second approver from a different function. This is general information, not legal advice. Employee monitoring raises questions of employment law as well as privacy, and each organisation should take its own advice.
The staff notice had an unexpected benefit. Once people knew that layer 4 was off by default for ordinary use, they were noticeably more willing to use the approved assistant rather than a personal tool, which is the outcome part 4 of this series is after.
Make the logs usable
A log nobody can query is not evidence. Fynbos applied four rules:
- One identity across layers. Every layer uses the same person identifier and entitlement ID, so a single query can join who was entitled, what they used and what it touched.
- Classification travels. Documents retrieved, files read and data products queried carry their classification into layer 3, so "did anyone touch restricted data?" is a filter, not a forensic project.
- Tamper evidence. Logs are written to a store that the users, and the platform administrators, cannot alter.
- Tested questions. Each quarter, the security team runs three or four investigation questions against the logs, like the mandate question, and records how long each took. Questions the logs cannot answer become backlog items.
Had Fynbos run that last test a year earlier, it would have found the coding-agent gap before a real investigation did. The test is cheap, and it turns logging from a configuration exercise into a tested capability. The same idea underlies Your agent spent the money. Can you prove why?: a record is only evidence if it can answer the question someone will actually ask.
Common mistakes
| Mistake | Why it fails | Better practice |
|---|---|---|
| Capturing every prompt and response by default | Creates a large, sensitive copy of data that was protected elsewhere | Capture content only where a stated purpose needs it |
| Logging usage counts only | Answers procurement questions, not investigation ones | Record actions: documents, commands, files, destinations |
| Different identifiers in each tool | No single query can join entitlement, session and action | One person ID and entitlement ID across all layers |
| One retention period for everything | Content is kept as long as metadata | Retention set per layer, content shortest |
| Never testing the logs | Gaps are found during a real investigation | Quarterly test questions with recorded answer times |
Key points
- Decide which questions the logs must answer before deciding what to capture.
- Use five layers: entitlement events, session metadata, action records, content capture and reliance records.
- Capture entitlements, sessions and actions always. Capture content only where a stated purpose requires it.
- For coding agents, the action layer (commands, files, network, packages) matters most; version control already holds the code.
- Logs about staff are personal information: publish what is logged, set retention per layer, restrict content access and test the logs with real questions each quarter.
Next: Part 8, How to restrict a tool without killing learning. All parts are on the series page. For the difference between logs, metrics and traces, see Logs, metrics and traces: what each one tells you when production breaks.
How CloudNala can help
CloudNala designs AI usage logging for regulated organisations. We start from the questions your investigation, risk and privacy teams need answered, define the five layers per capability, set retention and access rules with your information officer, and run test investigations against the result, so that the next difficult question has an answer within the hour.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Access Review or write to us at consult@cloudnala.co.za