Software has an established answer to "did that change break anything". You run the tests. The answer arrives in minutes, it is the same answer for everyone, and nobody argues about it in a meeting.
AI systems have the same problem and, in most organisations, no equivalent answer. A prompt is changed, a model is upgraded, a document is added to the index, and the way anyone finds out whether things got better is that a few people try it and report an impression. Impressions are heavily influenced by tone. An agent that has become more articulate and slightly less accurate will be reported as improved.
We have covered what evaluation means for agents in general terms. The argument here is narrower: the eval set has to be wired into the release path, not consulted alongside it.
The gate is the part people skip
Most teams that adopt evals adopt the first four boxes. There is a scenario set, it gets run, there are criteria, and there is a score. The score gets looked at.
The gate is what turns that from a habit into a control. A habit is "we run the evals and if something looks bad we investigate". A control is "the deploy does not proceed on a red result". The difference matters most exactly when it is most inconvenient, which is on the Thursday afternoon when the change is needed and the score dropped two points and everyone is fairly sure it is noise.
It usually is noise. The point of the gate is that "fairly sure" is not a decision-making standard, and that the alternative is a slow erosion where every individual override was reasonable and the cumulative effect is a system nobody trusts.
What a check actually looks like
Abstract discussion of evaluation makes this sound harder than it is. Five rows, for a tender compliance agent.
Each of these is a question with a known answer, tested against a document where somebody has already established what the right answer is. That is the whole mechanism. There is no need for a sophisticated judging framework to start; there is a need for twenty real documents and a note of what the correct output is for each.
The right-hand column is worth dwelling on, because it is how you get people to build these. Nobody is motivated by "we should have an eval suite". People are quite motivated by "last quarter we submitted a bid missing a compulsory attachment, and this check would have caught it".
The failure modes that matter most are silent
Three of the five checks above test for things the agent does confidently and wrongly, which is deliberate.
An agent that fails loudly is a manageable problem. It errors, someone notices, someone fixes it. An agent that cites a clause that does not exist has produced something that looks exactly like a correct answer and will pass casual review, particularly by a reviewer who is busy and has found the last twenty outputs to be fine.
This is why "does it escalate when unsure" belongs in the suite as a first-class check. The behaviour you want is not just accuracy, it is calibrated confidence: right when it says it is right, and explicitly uncertain when it is not. That property is testable, and it degrades silently when a prompt changes, which is precisely why it needs a test.
What triggers a re-run
Anything that can change behaviour, which is a longer list than most teams maintain:
- The prompt or workflow spec
- The model, including a minor version bump you did not choose
- The example library, since adding examples changes behaviour
- The document index or knowledge sources
- Retrieval settings
- Tool definitions and their descriptions
- Memory rules
Model version changes deserve particular attention because they arrive without your involvement. A provider updates a model, the behaviour shifts slightly, and the first indication is a user reporting that something feels different. If the suite runs on a schedule as well as on change, you find out first.
Building the first one this week
The barrier to entry is much lower than the discussion around evals suggests.
Take twenty real cases you already have. For each, write down what the correct output is, which is usually already recorded somewhere because a person reviewed it at the time. Run them. Count how many pass. That number is your baseline, and it is more useful than any amount of further discussion.
Then add a case every time something goes wrong. A missed requirement in production becomes case twenty-one. A confidently fabricated clause becomes case twenty-two. Within a few months the suite encodes your organisation's actual failure modes rather than a generic list, and that is when it becomes genuinely valuable.
The judging can start manual. Automating the comparison is an optimisation, not a prerequisite, and teams that wait until they have an automated judging framework often never start at all.
What good looks like
Twenty real cases with known answers, growing by one every time production surprises you.
Checks that include calibration, not only correctness.
A baseline number that everyone knows and that moves visibly.
Automatic re-run on prompt, model, data, example, retrieval, tool and memory changes, plus a scheduled run to catch provider-side drift.
A gate with an owner, where overriding a red result is a decision somebody makes on the record rather than a thing that happens.
Practical checklist
- Collect twenty real cases with known correct outputs
- Include at least one check for escalating under uncertainty
- Establish the baseline pass rate before changing anything
- Add a case for every production failure, with the document that caused it
- Re-run on every change to prompt, model, data, examples, retrieval, tools or memory
- Schedule a run to catch model updates you did not initiate
- Make the release gate real, and name who can override it
How CloudNala can help
We usually build the first eval set out of material the client already has, because the cases and their correct answers are sitting in review records and post-mortems. The engagement is less about tooling than about getting the failure modes written down as testable checks, and then putting the result somewhere it blocks a release rather than somewhere it gets read.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za