By the end of its rubric review, Kingfisher Contact Solutions had named owners, rewritten rubric lines, a calibrated model judge, a versioned rubric and a habit of running harness experiments. It had also discovered that most of this was held together by spreadsheets, one engineer's scripts and a weekly meeting that kept being cancelled.
The obvious next step was to buy an evaluation platform. Several vendors offered one. The team lead asked a different question first: what is the smallest set of parts that would make the work repeatable, and could they build it in a week? The answer turned out to be yes, and the week taught them which platform features they actually needed.
CloudNala's view is that evaluation capability starts as a small lab, not a platform: real cases, a versioned rubric, a runner, cheap code checks, a calibrated judge, a human review queue, a results table and a weekly review that is allowed to change the cases and the rubric. Build that first. Buy tooling once the lab has shown you which questions it needs to answer at scale.
This is the final part of a six-part series. Part 1 argued that whoever writes the rubric shapes the agent. Parts 2, 3, 4 and 5 covered writing, calibrating, versioning and experimenting. This part puts them into one working system.
Kingfisher Contact Solutions is fictional, and its systems and figures are illustrative. It is a Durban contact centre whose AI agent-assist drafts replies for three fictional client brands, and whose engineering team runs a coding-agent pilot.
Seven parts and a meeting
Case store. A folder or table of real cases, each with an input, any context the agent needs, the expected outcome where one exists and labels: brand, call type, difficulty, whether it is in the golden set or the hidden held-out set from Part 4. Kingfisher's lives in the same repository as the rubric. Customer details are removed before a case is stored, and the store holds only what the evaluation needs. The memory and POPIA guidance in an earlier series applies equally here: an evaluation set built from customer conversations is personal information until it has been properly de-identified.
Rubric repository. The versioned rubric from Part 4, with its layers, anchors, gate types, N/A rules and evidence requirements.
Runner. A script that takes a case set and a version of the agent, runs every case, and stores the outputs with the agent's version, the rubric version and a timestamp. For the coding-agent experiments in Part 5, the runner also restores the repository snapshot for each task.
Code checks. The deterministic checks from Part 3: required wording, format, facts against the source system, builds and tests for code. They run first because they are cheap and exact.
Model judge. The calibrated judge, applied only to rubric lines that have passed the human agreement test, and only to outputs that passed the code checks. It must return a quote as evidence for each score.
Human review queue. Outputs that need a person: the weekly audit sample, every must-pass failure, every case where the judge gave no evidence quote, and every case marked high-stakes. Reviewers see the output, the rubric line, the judge's score and quote, and record their own decision.
Results table. One row per output per rubric line, with the score, who or what produced it (code check, judge model and version, or reviewer), the evidence and the versions of the agent, rubric and case set. Everything else, including reports, is a query on this table.
The weekly review. A 45-minute meeting with the rubric owners, the quality lead and one engineer. It looks at three things: lines where the judge and the reviewers disagreed, cases that failed in a new way, and any proposed rubric changes. It is the only place where cases are added to the set or rubric changes are approved, which keeps both under the ownership rules from Part 1.
None of these parts needs specialised software. Kingfisher built the first version with a repository, a small database, the model provider's existing interface and a review form.
Five days to a working lab
Day 1: collect 50 real cases. Pull recent outputs that represent the real mix of work, including some known failures. Remove personal information. Resist the urge to collect hundreds: 50 well-chosen, well-labelled cases are more useful in week one than 500 unlabelled ones. Add the failures from complaints or incidents first; they are the cases the lab most needs to catch.
Day 2: write five rubric lines. Not the whole rubric. Choose the must-pass items and the two or three weighted lines that matter most, and write them using the eight parts from Part 2, with anchors taken from the Day 1 cases. Have each line's owner approve it.
Day 3: two people label by hand. Two people score all 50 cases on all five lines, independently. Calculate agreement per line. Rewrite any line they cannot agree on, then label again. By the end of the day, you have a small golden set with agreed labels.
Day 4: wire the runner, checks and judge. Build the runner, implement the lines that code can check, and point the model judge at the remaining lines. Run it on the 50 cases and compare its results with the agreed human labels. Where it disagrees, look at its evidence quotes.
Day 5: first weekly review. Hold the meeting. Look at the disagreements, decide whether each is a judge problem, a rubric problem or a case problem, and agree what changes. Schedule the next one.
At the end of the week, the lab is small but honest. It can tell the team whether a change to the agent, the prompt or the harness made things better on cases that matter, with a known level of agreement between the judge and people. A larger tool that cannot report that agreement is giving you less, not more.
What to leave for later
Three things are tempting in week one and better left alone.
A dashboard. Dashboards are easy to build from the results table, and they are useful once you know which questions you ask every week. Built in week one, they display whatever is easiest to count. Kingfisher's first dashboard, built in the lab's second month, had four charts, each one chosen because the weekly review had asked for it more than twice.
A judge model of your own. Fine-tuning a dedicated judge can make sense at very high volume, but only once you have a large set of agreed human labels to train and test it on. In week one, you have 50.
A platform purchase. Evaluation platforms offer case management, judges, dashboards and review queues, and some are good. Buying one before running the lab tends to mean adopting the platform's defaults, including its default criteria, which is the problem Part 1 started with. After a few weeks of the lab, you will know which features you need and can judge the platforms against your own requirements.
Growing the lab without losing its discipline
Kingfisher's lab has grown over its first two months. The case set now has about 600 cases across the agent-assist and the coding-agent pilot. The rubric has three layers and 19 lines. The weekly review still takes 45 minutes.
Three rules have kept it useful as it grew.
Every incident becomes a case. When a complaint or escalation involves the agent, the conversation, de-identified, joins the case set, labelled with the failure it represents. The lab's cases become a record of what has gone wrong, which is the same principle as building a golden set from reality.
Every result carries its versions. Agent, rubric, judge and case set, on every row. This is what makes the drift questions in Part 4 cheap to answer.
Only the weekly review changes the definition of good. Engineers can add cases to their own working sets, but the golden set, the hidden set and the rubric change only through the review, with the owner's approval.
The evals that gate real work article describes how evaluation results should connect to release decisions. Kingfisher's lab now feeds exactly that: no prompt, model or harness change reaches production without a run against the golden set, and the results go into the change record.
Who should own the definition of good
This series began with a contact centre whose AI agent had quietly learned to write long, warm, generic replies because that is what an unowned rubric rewarded. It ends with a small lab that makes the definition of good explicit, versioned, calibrated and owned.
CloudNala's view is that the rubric is where an organisation's intent meets its AI systems, and it deserves the same seriousness as a policy, a contract or a release process. That means the people accountable for an outcome decide what good looks like, the people with the craft write it down so it can be scored, the people who run the system measure it honestly, and nobody gets to change the definition of success because it makes their own numbers look better.
The tools for this are modest. The discipline is not, and it is the difference between an agent that does what the organisation wants and one that does what the scoring happened to reward.
Start this week
- Block five days for one engineer and one quality lead, with an hour a day from each rubric owner.
- Follow the day-by-day plan above, and keep the case set at 50 for the first week.
- Book the weekly review for the next eight weeks before Day 1, so it does not get squeezed out.
- List the platform features you think you need, then revisit the list after four weekly reviews.
That completes the series. Start again from Part 1, Whoever writes the rubric shapes the agent, or see all parts on the series page.
How CloudNala can help
CloudNala helps organisations build their first evaluation lab in a week. We work alongside your team to assemble real cases, write and calibrate the first rubric lines with their owners, build the runner, code checks and judge, and run the first weekly reviews with you. When the lab has shown what you need at scale, we give you an independent view on whether to extend it or buy a platform, and against which requirements.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za