Ask most teams running an AI pilot how well it works and you will get an answer built from impressions. It is good at summaries. It sometimes gets confused with the older documents. People seem to like it.
None of these are measurements, and their absence has a specific consequence: the team cannot tell whether a change helped. Someone adjusts a prompt, the outputs feel different, and there is no way to establish whether accuracy improved, declined, or moved in one area at the expense of another. Development becomes a sequence of confident guesses.
Evaluation is what converts this into engineering.
The golden set
The foundational artefact is unglamorous: a list of questions or cases where you already know what the right outcome is.
Thirty is enough to start. They should come from the actual work — questions users genuinely ask, documents that genuinely arrive — and the correct answers should be agreed by someone who knows the material well enough for their judgement to stand.
Deliberately include the hard cases. The question the documents do not answer, where the correct behaviour is to say so. The document with a scanning error. The request that should be escalated rather than handled. The edge case that trips up new staff members, because it will trip up the agent for similar reasons.
Once this set exists, "we changed the prompt" stops being a leap of faith. You run it, you compare, you know.
What to actually check
For a retrieval system, four checks cover most of the ground: was the answer factually correct; did it cite a real source that genuinely supports the claim; did it correctly refuse when the material was not there; and did it avoid disclosing anything the asking user should not see.
The citation check deserves emphasis because it is frequently done badly. It is not enough that a citation is present. The cited section has to actually say what the answer claims it says. A system that cites confidently and inaccurately is more dangerous than one that does not cite at all, because it manufactures the appearance of verification.
For an agent, add three more: did it select the right tool; did it call it with valid arguments; and did it escalate when it should have. That last check is the one teams skip, and it is the one that governance committees ask about.
Four ways to judge
Deterministic checks are the best available option whenever the question permits them. Did it extract the correct reference number? Did it classify this document as a tender rather than an invoice? Did it call the compliance tool? These are exact comparisons, they run in seconds, they cost nothing, and they never disagree with themselves. Use them wherever the answer has a defined correct form.
A model as judge handles the questions deterministic checks cannot — is this summary faithful to the source, is this tone appropriate, does this answer address what was asked. A second model, given the source material and clear criteria, scores the output. It is cheap and it scales, and it is imperfect: it tends towards generosity, and it can share the blind spots of the model that produced the answer. Calibrate it against human judgements on a sample before trusting it, and re-calibrate periodically.
Human review on a sample remains necessary. Not on everything — that defeats the point — but a structured review of a random selection each week, by someone who knows the domain. This catches the failures that are obvious to an expert and invisible to any automated check.
Business outcomes are what the organisation actually cares about. Cycle time before and after. Rework rate. How often a human overrides the output. How many cases reached the right team first time. These are lagging, noisy and slow, which is why the other three exist — but they are the ones that determine whether the system continues to be funded.
The metric that matters most
Track one number above the others: how often the system is confidently wrong.
An AI system that says "I do not know" is a system with a usability problem. An AI system that produces a fluent, well-formatted, entirely incorrect answer with a plausible citation is a system with a liability problem. These are not the same failure and they should never share a metric.
Splitting accuracy into correct, honestly-uncertain and confidently-wrong changes how teams behave. It makes refusal a legitimate outcome rather than a failure, which removes the incentive to tune the system towards always producing something. And it gives risk committees the number they actually want, which is not "how accurate is it" but "how often could it mislead someone".
Examples worth stealing
For a tender assistant, the evaluation set should assert that it cites the specific clause for every requirement it states; that it never invents a requirement that is not in the documents; that it identifies missing documents against a known checklist; and that it escalates rather than answering when a question turns on legal interpretation.
For a citizen-facing service assistant, the checks are that it never states policy that does not exist in its sources; that it classifies request types in line with the department's own taxonomy; that safety-critical reports are escalated immediately regardless of anything else in the request; and that responses stay within the reading level and bandwidth the service commits to.
Both sets share a structure: some checks are about being right, and some are about knowing when not to answer. The second kind is what makes the system trustworthy.
When to run it
Before every change to the prompt, the model, the retrieval configuration, the tool definitions or the source data. All five of those alter behaviour, and the fourth and fifth surprise people.
Also on a schedule, without any change at all. Model providers update their models. Documents get revised. Assumptions that held in March stop holding in September. A monthly run against an unchanged system is how you find out that the ground moved.
How to start small
Write ten test cases this week. Not a hundred — ten, covering the most common question, the most dangerous failure, and one case where the correct answer is "I do not know". Run them by hand and write down the results in a spreadsheet.
That spreadsheet is a legitimate evaluation suite. Automate it when the manual run becomes annoying, which will be at around thirty cases. Add a case every time the system gets something wrong in real use — this is the habit that compounds, because your test set then grows in exactly the directions your system is weak.
Within a few months you will have something that lets you change the system confidently and tell a governance forum precisely how often it is wrong and in what way. Very few organisations running AI pilots can do that today.
How CloudNala can help
We build the evaluation suite alongside the system rather than after it — golden sets drawn from your real cases, deterministic checks wherever the answer has a defined form, calibrated model-as-judge scoring where it does not, and a confidently-wrong metric reported separately from ordinary accuracy. It is what turns an AI pilot into something a risk committee can sign off, and it is usually the fastest route to getting a stalled pilot approved.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za