Evals that gate real work

Evaluation starts by defining what "good" means for a particular task, user and level of risk.

An FAQ answer, a clinical summary, a credit recommendation, a marketing draft and an infrastructure change cannot share one universal quality score. The consequence of failure differs. The available evidence differs. The person entitled to accept the result differs. An evaluation that ignores those differences produces a number, and a number on its own changes nothing.

Part 2 made the distinction this part depends on: a trace records what happened, and an evaluation judges it against a defined standard. This part is about the judging, and about the step most teams skip, which is deciding in advance what each result should cause to happen.

Weaverbird Agri is fictional, and its systems and figures are illustrative. Weaverbird's Scout agent writes weekly commodity and competitor briefs for the commercial team: around 30 a month. Scout is the agent that once reported "no competitor promotions found on social media" when the social-search tool had actually failed (Part 3). Its evaluation design is what stops that happening again.

Five complementary forms of evaluation

No single method covers everything. Mature teams combine five, and use each for what it does well.

1. Deterministic checks

Code verifies properties that have a clear answer:

• required fields are present;
• a total matches its line items;
• each citation resolves to evidence that was actually retrieved;
• exactly one report was created;
• the run stayed within budget;
• no forbidden tool was called.

These checks are cheap, repeatable and easy to explain. Use them wherever a requirement can be expressed in code. A surprising share of the most important requirements can be.

2. Rule and policy checks

The system tests the output, or a proposed action, against business, security or regulatory rules: personal information is not exposed, a credit above a threshold requires approval, advice carries the required disclaimer, the agent did not act outside its delegated authority.

Version the rules. Months later, the organisation must be able to reconstruct which rule applied when a decision was made.

3. Model-based evaluation

A second model judges qualities that are hard to express in code: relevance, clarity, or whether an answer is actually supported by the evidence supplied.

Model judges are useful, and they have known weaknesses. They can be inconsistent, they tend to favour fluent style, and they can be manipulated by text planted in the content they are judging. They need written rubrics, representative test data, version control, and regular comparison with human judgement. An uncalibrated judge is a second opinion from someone you have never checked.

4. Human evaluation

Specialists assess ambiguous cases, high-impact decisions, and situations where context matters more than any score.

Human review is not automatically right either. Reviewers need guidance, sampling plans, a way to resolve disagreement and feedback on their own consistency. Capture why a reviewer overrode the agent: that reason is often the most valuable evaluation data you will collect.

5. Business outcome evaluation

The strongest signal often arrives after the answer has been delivered. Was the case resolved? Did the user accept the output or correct it? Did rework fall? Did the recommendation lead to a compliant decision? Did conversion improve without more complaints?

Outcome measures are slow and noisy, so they cannot gate an individual answer. They are what tells you whether all the faster checks are measuring the right things.

An evaluation portfolio, not a single test

Offline evaluation. Before any release, run a fixed, versioned dataset: normal cases, difficult cases, known failures, malicious inputs, missing data and tool outages.

Shadow evaluation. Run the candidate on real traffic without letting it affect the user or the business process, and compare it with the current version.

Canary release. Expose a limited, controlled share of traffic and watch quality, safety, cost and operations before expanding.

Online evaluation. Assess production runs continuously with deterministic rules, sampled model judges, human review and business outcomes. This is what catches drift and conditions the original test set never imagined.

When Weaverbird's platform team wanted to move Scout's synthesis step to a newer model version, the change went through all four. It passed offline evaluation, then ran in shadow for two weeks, writing briefs that nobody received, so the digital desk could compare them with the live ones. Only then did it go to a canary. Continuous evals covers the online half in more depth.

A worked scorecard

Here is the scorecard Weaverbird uses to release a Scout brief.

Not every failed measure should block the user. A latency breach triggers an operational alert while an accurate brief still goes out. A missing mandatory source blocks a research conclusion. The release effect is part of the definition of the evaluation, decided in advance, not argued about after the fact.

The first two rows are the direct legacy of the Part 3 failure. "Required sources completed" reads the trace rather than the prose, so a brief cannot claim a finding from a source that never returned. The R50 threshold covers the model, tool, memory and evaluation spend for one brief. Human review time is tracked separately, because it is by far the largest part of the R103.60 total cost from Part 1.

Build the golden set from reality

A useful evaluation set contains:

• common tasks that reflect real demand;
• edge cases and rare but severe failures;
• adversarial and prompt-injection cases;
• different languages, writing styles and user groups;
• missing, conflicting and outdated evidence;
• tool timeouts, permission failures and malformed responses;
• known historical incidents;
• cases where the right answer is to refuse or escalate.

Weaverbird's golden set for Scout has about 120 cases, built from real past briefs rather than invented examples. It includes the social-search failure from Part 3, replayed with the tool forced to fail, so the test checks that the brief says the source was unavailable instead of reporting zero results. Depot Desk's set includes the stale-memory case from Part 4: a dealer whose credit terms changed after the agent first learned them. Every incident adds at least one case. That is how a golden set gets better instead of merely bigger.

Protect the set from contamination. If the examples leak into prompts, few-shot libraries or training data, scores can rise without the production system improving. Keep the set versioned and access-controlled, and treat a sudden jump in scores as a reason to check for leakage before celebrating.

Test the whole system

Evaluating the model on its own misses failures created by retrieval, tools, orchestration, identity and the user interface. At minimum, evaluate:

• model behaviour;
• retrieval relevance and permissions;
• tool selection and the parameters used;
• loop termination;
• disclosure of errors and uncertainty;
• policy enforcement;
• cost and latency;
• end-to-end task success;
• human escalation;
• business outcome.

Most of the incidents in this series would have passed a model-only test. The social-search failure was a tool and disclosure problem. The stale credit terms were a memory problem. The duplicate draft credit note in Part 5 was a retry problem. None of them was the model writing badly.

The release decision

An evaluation score should connect to an owner and an action.

At Weaverbird, "evidence incomplete" on a Scout brief means the brief is held, with a coverage line explaining what failed, and the Head of Commercial's team decides whether a limited brief is still useful that week. For Depot Desk, a failed authority check on a credit note request means the request never reaches the approval queue at all.

The standards bodies point the same way. The NIST AI Risk Management Framework, and its Generative AI Profile (NIST AI 600-1), treat measurement as something that must feed management decisions, not a report on the side. The OWASP Top 10 for LLM Applications (2025 edition) and its newer Top 10 for Agentic Applications are good sources of adversarial cases for a golden set. None of them gives you thresholds; those come from your own risk appetite. For the broader question of whether an agent is working at all, see evaluating AI agents.

The short version

Define good per task. One universal quality score does not exist.

Use all five forms. Deterministic checks wherever possible, calibrated model judges where necessary, people where it matters, and outcomes to keep everything honest.

Release through a portfolio. Offline, shadow, canary, online: every material change, every time.

Grow the golden set from incidents. Every failure becomes a test case.

Wire each result to an action and an owner. An evaluation only becomes a control when its result changes what the system, or an accountable person, does next.

Previous: Part 5, the gateway is a control point. Next: Part 7, from agent dashboard to operating model. All parts are on the series page.

How CloudNala can help

We help teams turn "it looks good" into a release decision: writing task-specific scorecards with your domain owners, building golden sets from your own past work and incidents, calibrating model judges against human review, and wiring each result to a defined action and owner. We also set up the shadow and canary steps, so that model and prompt changes are released on evidence rather than on optimism.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agent Evidence Review or write to us at consult@cloudnala.co.za