Kingfisher Contact Solutions' agent-assist produces about 40,000 draft replies and after-call summaries a week. Its quality team can review a few hundred. So, like most organisations running an agent at volume, Kingfisher uses a language model to grade the agent's output against the rubric. The model reads each draft, applies each rubric line and returns a score.
The scores arrive with decimals, in tidy tables, every morning. They look more precise than the human audits ever did. The question Kingfisher had never asked was whether they were right.
CloudNala's view is that a model judge is an instrument, and an instrument has to be calibrated against a reference before its readings mean anything. The reference is agreed human judgement, so the people have to be calibrated first. A criterion that two trained people cannot agree on cannot be delegated to a model, however confident the model sounds.
This is Part 3 of a six-part series. Part 2 showed how to write rubric lines that can be scored. This part covers how to check that they are being scored correctly. Earlier CloudNala articles describe four ways to judge an agent's output and how evals gate real work; this part goes deeper into one of those ways, the model judge, and what it takes to trust one.
Kingfisher Contact Solutions is fictional, and its systems and figures are illustrative. It is a Durban contact centre whose AI agent-assist drafts replies and after-call summaries for three fictional client brands.
What the research says about model judges
Model judges are not a fringe technique. A widely cited study, Zheng and colleagues' 2023 paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", found that strong models used as judges could reach over 80% agreement with human preferences, which the authors note is the same level of agreement found between humans. That is a useful result (paper checked on 31 October 2026).
The same paper also examined the judges' limitations, including three biases that every team using a model judge should design around:
- Position bias: when comparing two answers, the judge tends to favour one position, for example whichever answer is shown first.
- Verbosity bias: the judge tends to favour longer answers, even when the extra length adds nothing.
- Self-enhancement bias: the judge may favour answers produced by itself or by models like it.
The paper also notes limited reasoning ability when grading maths and reasoning questions. Two points follow. First, "over 80% agreement" was measured on that study's tasks and criteria, not on yours; it tells you a judge can work, not that yours does. Second, the biases point in a familiar direction. Kingfisher's agent-assist had drifted towards long, warm replies, and a judge with a verbosity bias would reward exactly that drift.
Calibrate the people first
The natural first step is to compare the model judge with a human grader. That skips a step. If the human graders do not agree with each other, there is no stable reference to compare the model against.
Kingfisher's calibration process now runs in this order.
Sample real outputs. 200 drafts drawn from the last month across all three brands, stratified so that unusual call types are represented, with customer details removed.
Two people label each one independently. Both are experienced quality analysts who have been briefed on the rubric and the anchors from Part 2. Neither sees the other's labels.
Measure human agreement per line. Kingfisher uses Cohen's kappa, a statistic introduced by Jacob Cohen in 1960 that measures agreement between two raters after correcting for the agreement you would expect by chance. Raw percentage agreement is misleading on lines where almost everything passes: if 95% of drafts pass, two graders who mark everything as a pass agree 95% of the time without looking. Kappa corrects for that.
Kingfisher set its own threshold. A line moves on to the model judge only if its human kappa is at least 0.6. That number is a working choice, not a standard; what matters is that it was set before looking at the results. On the first run, "Names the customer's actual problem" reached 0.74. A separate line on "appropriate tone", added to Harbour Home's layer alongside the voice line from Part 2, reached only 0.38.
If people disagree, rewrite the line. The tone line went back to its owner with the disagreements attached. The analysts' disagreements were concentrated on replies that used humour, which the brand guide neither encouraged nor banned. The owner decided, the anchors were updated and the line passed on the second run.
Then measure the judge against the people
Only lines that pass the human test are given to the model judge. The judge then scores the same 200 drafts, and its scores are compared with the agreed human labels, line by line. Again Kingfisher uses kappa, and again it sets the threshold before looking.
The line-by-line view matters. A judge that agrees with people on average can still be poor on one line. Kingfisher's judge matched the analysts well on "Says what happens next, and when", because the line has a clear textual signal. It was weaker on Aloe Cover's line about explaining waiting periods correctly, because judging that requires knowing the product rules, which the judge did not have. That line moved to a different kind of check, covered below.
Kingfisher also checks for the three published biases directly:
- Position: wherever the judge compares two drafts, it runs each comparison twice with the order swapped and counts only results that agree.
- Verbosity: it takes a set of passing drafts, has the agent produce padded versions that add length but no information, and checks whether the judge's scores rise. They did, slightly, on two lines, which were rewritten to say that unnecessary length counts against the draft.
- Self-enhancement: the judge is a different model family from the one that drafts the replies, so the agent is not grading its own family's style.
Requiring evidence, as Part 2 recommends, makes all three easier to catch. When the judge must quote the sentence that justifies each score, a reviewer can check a judgement in seconds, and scores without a supporting quote are flagged automatically.
Choose the checker by the kind of criterion
A model judge is not the right checker for every line. The calibration work at Kingfisher ended up sorting its rubric into six kinds of criterion, each with a different checker.
Exact wording or format is checked by code. Aloe Cover's waiting-period disclosure has required wording; a pattern match finds it or does not. Using a model here adds cost and uncertainty for no gain.
Facts against a record are checked by code against the source system. If a draft says the technician is booked for Thursday, the check reads the booking system. A model judge cannot see the booking system and should not guess.
Tone, clarity and relevance are where a calibrated model judge earns its place. These need language understanding, the volume is far too high for people, and the lines have passed the human agreement test.
Compliance must-pass items get code checks first wherever possible, and a person reviews every failure before it is reported to a client. A false failure on a compliance line is expensive, and so is a false pass.
Novel or high-stakes cases go to a person. Kingfisher flags calls involving a bereavement claim, a complaint to the ombud or a threat of legal action, and these are reviewed by an analyst, not graded automatically.
A criterion people disagree on is checked by nobody until it is rewritten. Grading it automatically would only produce confident noise.
This sorting changed the economics as well as the quality. Code checks are cheap, so running them first and sending only passing drafts to the model judge cut Kingfisher's judging cost while making the important checks more reliable.
Keep an audit sample running
Calibration is not a one-off event. Three things change over time: the agent's outputs, the judge (vendors update models, sometimes under the same name) and the kinds of calls coming in. Any of them can quietly break the agreement measured at launch.
Kingfisher keeps a standing audit sample. Every week, 100 drafts the judge has already scored are re-scored by an analyst, chosen partly at random and partly from cases where the judge's evidence quote was missing or its score sat on a threshold. Agreement is tracked per line. If agreement on a line falls below its threshold for two weeks running, the line goes back through calibration.
The judge's model name and version are stored with every score. When the provider announces a model update, Kingfisher re-runs the 200-draft calibration set before switching, and treats a significant change in agreement as a reason to delay the switch. Part 4 covers why every score must carry this kind of version information.
Start this week
- Pick one rubric line your model judge scores today. Have two people label 50 to 100 real outputs on that line independently, and calculate their agreement with kappa as well as raw agreement.
- If people agree, compare the judge with their agreed labels. If people do not agree, stop: rewrite the line first.
- Test for verbosity bias by padding a few passing outputs and checking whether the judge's scores rise.
- Sort your rubric lines into the six kinds above, and move anything code can check exactly out of the model judge.
- Store the judge's model and version with every score, and set up a small weekly audit sample.
Next: Part 4, Version the rubric like code. All parts are on the series page.
How CloudNala can help
CloudNala helps teams calibrate their evaluation before they rely on it. We design human labelling rounds on your own outputs, measure agreement line by line, test your model judge for the published biases, decide which criteria belong with code, with a judge or with a person, and set up the audit sampling that keeps the measurements honest after launch.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za