In August, Kingfisher Contact Solutions' quality score for Coastlink dropped by eleven percentage points in a single week. The engineering team spent two days looking for what had broken in the agent-assist. They compared prompts, checked the model provider's status page and pulled sample drafts. The agent was fine.
What had changed was the rubric. A team lead had tightened the anchors on "Says what happens next, and when" after a client complaint, and saved the file. Nobody had told engineering, the scores were not tagged with a version, and the weekly report compared the new week against old weeks scored under different rules.
CloudNala's view is that a rubric is code that defines success, and it needs the same change control as code: a version number, a reason for every change, an approval from the owner, a test against a fixed set before release, and a version tag on every score it produces. Without that, a rubric change looks exactly like a change in the agent, and teams waste time fixing things that were never broken.
This is Part 4 of a six-part series. Part 1 gave the rubric an owner, Part 2 showed how to write it and Part 3 how to calibrate the judge that applies it. This part is about changing it safely.
Kingfisher Contact Solutions is fictional, and its systems and figures are illustrative. It is a Durban contact centre whose AI agent-assist drafts replies and after-call summaries for three fictional client brands.
The rubric lives in a repository
Kingfisher's rubric used to live in the evaluation tool's settings screen, where anyone with admin access could edit it. It now lives as text files in a repository, alongside the prompts and the test cases: one file per layer (the shared must-pass floor, the Kingfisher service standard and each brand's own lines), with the anchors, examples, gate types and N/A rules from Part 2 written out in full.
That move alone fixed three problems. Every change now has an author, a date and a diff. The evaluation runner reads the rubric from the repository at a known version, so engineering can see exactly which rules produced a score. And the brand owners can read their own layer in plain text, without needing access to the tool.
The repository does not need to be sophisticated. What matters is that the rubric has a single source of truth, a history and a version that the scoring system records.
A rubric change travels the same path as a code change
Every change now follows the same five steps.
Change request with a reason. Whoever wants the change writes down what should change and why, usually with examples: the drafts that were scored wrongly under the current wording.
Rubric owner approves. The named owner for that layer, from Part 1, approves or rejects it. For the must-pass floor, that is compliance. For a brand's layer, it is the brand owner. The engineer who will benefit from a higher score never approves.
Re-score the golden set. The golden set is a fixed collection of real outputs, with customer details removed, that has been scored and agreed by people. Kingfisher's has 300 drafts across all three brands. Before release, the runner scores it under both the old and new versions of the rubric.
Explain every moved score. Any draft whose result changes between versions is listed, and the requester explains why the new result is right. If a change was meant to affect one kind of draft and moves scores on others, that is a sign the wording has side effects.
Release and tag all scores. The new version goes live, and every score from that point carries the rubric version alongside the judge's model and version from Part 3.
The Coastlink incident would have looked very different under this process. The tightened anchors would have moved scores on the golden set, the team lead would have explained the expected drop, and the weekly report would have shown a version boundary rather than a mystery.
Patch, minor and major
Not every change matters equally, and the version number should say which kind of change it was. Kingfisher borrowed the familiar three-part convention from software versioning.
A patch is a wording change that should not move any score: a typo, a clearer example of an existing anchor. If the golden-set re-score shows moved scores, it was not a patch.
A minor version adds or retires a criterion. Scores on the other lines remain comparable with earlier versions, but the overall total does not, so reports show the totals separately on either side of the change.
A major version changes a scale, the weights or what counts as must-pass. Scores under a major version cannot be compared with earlier scores at all unless the history is re-scored. Kingfisher keeps the raw outputs for this reason: if a trend line across a major version matters, it re-scores the earlier period under the new version rather than drawing a line across two different rulers.
The convention is less important than the habit. Every reader of a quality report should be able to see where the rulers changed.
Score drift is not behaviour drift
When a score moves, four different things could have changed, and only one of them is the agent.
Did the rubric change? Check the version tag. If it changed, compare like with like by re-scoring on one version.
Did the judge change? Check the judge's model and version. Providers update models, and a model judge's agreement with people can shift without any change to the rubric. If it changed, recalibrate as in Part 3.
Did the case mix change? In June, a fibre outage flooded Coastlink's queue with fault calls that no reply could resolve on first contact, and Coastlink's scores fell for two weeks. That was a change in the work, not in the agent. Compare the same call types across periods.
None of these? Then the agent's behaviour changed, and it is worth investigating: a prompt change, a model update on the agent side, a tool that started returning different data.
This order matters because the first three questions are cheap to answer if scores carry their versions, and expensive if they do not. The Agent Evidence Layer series makes a related point that a trace is not an evaluation; the corollary here is that an evaluation score without its version information is not evidence either.
When the agent learns the rubric
Part 1 quoted the line usually called Goodhart's law: when a measure becomes a target, it ceases to be a good measure. A versioned rubric does not repeal it. Teams still tune the agent to the scores, and over time the agent can learn the rubric's surface features rather than the behaviour the rubric was trying to describe.
Kingfisher saw an early sign of this. After "Says what happens next, and when" was introduced, nearly every draft began to end with a sentence of the form "You can expect an update within 24 hours". The line was passing. But for calls where nobody would in fact update the customer within 24 hours, the sentence was a promise the human agent then had to break.
Three practices help.
Keep a hidden held-out set. Alongside the golden set used for rubric changes, Kingfisher keeps a separate set of cases that engineers tuning the agent never see. If scores on the visible set rise and scores on the hidden set do not, the agent is learning the test. The Agent Evidence Layer series covers protecting a golden set from leakage; the hidden set applies the same idea to the people tuning the agent.
Spot-check against outcomes, not only scores. Each month, a sample of passing drafts is traced to what actually happened: did the customer call back, was the promised update sent? This is how the 24-hour promise was caught.
Tighten anchors with real failures. The fix looked small, but it changed what passes, so it went through the full process rather than as a patch: the anchor for the top level now requires the stated time to match the booking or ticket system. That moved the line from a model judge to a code check against the record, as in Part 3.
Retiring a criterion
Rubrics tend to grow. Every incident adds a line, and nothing is ever removed. A long rubric is slower and more expensive to apply, harder to calibrate and more likely to contain lines that contradict each other.
Kingfisher reviews each layer quarterly and asks of every line: has it failed anything in the last quarter, and does its owner still want it? A line that has not failed anything in months may be guarding against something that no longer happens, or it may be passing everything because it is too loose. Either way, it gets a decision. Retiring a line is a minor version, with the same re-score and explanation as adding one.
Start this week
- Move your rubric into version control as plain text, one file per layer or owner.
- Record the rubric version and judge version with every score from now on. If your tool cannot store it, store it beside the tool.
- Build a golden set of 100 to 300 real outputs your people have scored and agreed, and re-score it before any rubric change goes live.
- Write down your four drift questions and ask them, in order, the next time a score moves.
- Create a hidden held-out set that the people tuning the agent cannot see.
Next: Part 5, Stop debating harnesses. Run the experiment. All parts are on the series page.
How CloudNala can help
CloudNala helps teams put their evaluation under change control. We move rubrics into versioned repositories, design golden and held-out sets from your own data, add version tagging to every score, and set up the drift diagnosis that separates changes in the measurement from changes in the agent, so your teams fix what is actually broken.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za