Ask an organisation how it audited its AI system and the answer often describes two things: the model was assessed for bias or accuracy, and a responsible AI policy was signed. Both are worth doing. Neither tells you much about whether the system in production behaves as intended, whether anyone could reconstruct a disputed case, or what happens when a vendor quietly changes the model underneath.
An AI system is not a model. It is a model plus the data it retrieves, the instructions it is given, the tools it can call, the people who can make it act, the logs it keeps, the evaluations that justify trusting it, and the suppliers it depends on. An audit that examines only the model audits the least changeable part of the system and skips the parts that change every week.
This part sets out what a useful AI audit covers, framed for the people commissioning one: risk, internal audit, compliance and the executives who receive the report.
Silverleaf Brokers is fictional, and its systems and figures are illustrative. Its advisers draft client review packs with an AI assistant, as described earlier in this series. After a year in use, its board asked internal audit to review the assistant. The plan below is the one a sensible audit team would draw up.
Eleven areas in three groups
What the system is made of.
- Models and versions. Which models are used, through which provider or route, at which version, and how is a version change detected?
- Data and retrieval sources. What can the system read? Who decides what goes into the document store, how is it kept current, and does retrieval respect the same permissions as the source system?
- Prompts and instructions. Are system prompts, agent instructions and templates versioned, reviewed and owned, or edited ad hoc in a console?
- Tools and permitted actions. What can the system do, not just say? Which actions are read-only, which change records, which send anything outside the organisation?
How it is controlled.
- Access and identity. Who can use it, who can make it act, and does it act with the user's authority or with a shared service account?
- Evaluation evidence. What test set justifies trusting it, when was it last run, and does it include the hard cases or only the demo?
- Logs and traceability. Can a single disputed case be rebuilt: the input, the retrieved content, the prompt version, the model version, the output and the human decision?
- Human oversight. Where are humans in the loop, what do they actually check, and is there evidence they do?
What it costs and depends on.
- Incidents and near misses. What has gone wrong, how was it detected, and what changed afterwards?
- Cost and cost ownership. Who owns the bill, is cost tracked per use case, and are there limits?
- Vendor and dependency risk. What happens if a provider changes a model, retires a version, changes terms, or has an outage?
Audits commonly look hard at the first group, lightly at the second and barely at the third. Real incidents tend to originate in the second and third.
Question, evidence, common gap
An audit plan is only useful if each area turns into a question that can be answered with evidence rather than assurance.
A few of these deserve more explanation, because they are where Silverleaf's audit found its most important issues.
Evaluation measured on what. The assistant had been evaluated before launch on twenty example packs, all for straightforward personal lines clients. In production, a third of packs were for small commercial clients with fleets and multiple sites. There was no evidence at all that the assistant handled those well. The finding was not that it performed badly; it was that nobody knew. Evals that gate real work describes how to make the test set track the real mix.
Can one case be rebuilt. The system kept the final pack and the approval but not the retrieved schedule or the prompt version. When the audit team picked a disputed renewal at random, they could show what was sent, but not why it said what it said. The retention gap took a day to fix and would have taken months to explain in a complaint.
Who can make it act. The assistant updated the client record through a single integration account with broad write permissions. Any adviser could, through the assistant, change a record they could not change directly. It is a common pattern: the AI layer quietly widens access. Identity before intelligence covers the design pattern that avoids it.
Silent vendor changes. The assistant used a model alias that the provider updated twice during the year. Output style changed noticeably after the second update. Nobody had linked the change to the alias because the version in use was never recorded per case.
What the audit report should say
The output of an AI audit should be useful to two different readers.
For the board or audit committee: an overall view of whether the system is operating within the organisation's risk appetite, the three to five findings that matter most, and who owns each corrective action.
For management and the delivery team: each finding with its evidence, the risk it creates, and a recommendation specific enough to act on.
Two disciplines from ordinary audit practice matter especially for AI:
- A finding is not a corrective action. The auditor reports what was observed and the risk. Management owns the response. Auditors who design the fix put their independence at risk when they review it later. Our Technology Health Check series covers this in more depth on the series page.
- Avoid unsupported root-cause claims. "The model is biased" is rarely a supportable finding from an audit. "The evaluation set does not include commercial clients, so performance on them is unknown" is supportable and actionable.
How often, and how deep
AI systems change more often than most systems that get audited annually. Prompts, retrieval content and model versions can all move weekly. Two practical adjustments help:
- Continuous evidence, periodic judgement. Make the system produce the evidence continuously (evaluation runs, version logs, access reviews, incident records), so the periodic audit reviews a record rather than reconstructing one.
- Trigger reviews on material change. A new tool with write access, a new data source, a new model family, or a move into a new client segment should trigger a focused review, not wait for the annual cycle.
Proportion still applies. An internal assistant that summarises public documents does not need the same audit as one that drafts client advice or acts on records. Scope the audit by consequence.
Next: Part 5, A secure enterprise AI reference architecture, layer by layer. All parts are on the series page. For what traces can and cannot prove, see A trace is not an evaluation.
How CloudNala can help
We help risk, audit and technology teams scope and run AI system reviews that go beyond the model: building the audit plan across all eleven areas, requesting and reading the right evidence, and writing findings that management can act on. We can also help teams make their systems audit-ready by producing that evidence continuously.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Assurance Review or write to us at consult@cloudnala.co.za