Once Kingfisher Contact Solutions had named an owner for each decision about its rubric, it hit the next problem. The owners agreed on what they wanted. They could not write it down in a form that anyone, human or model, could score consistently.
The brand owner for Harbour Home wanted replies that "sound like us". Aloe Cover's compliance lead wanted replies that were "accurate and compliant". Coastlink wanted replies that "actually solve the problem". These are reasonable goals. As rubric lines they are useless, because two careful graders reading the same reply will give different scores, and neither can explain the difference in a way the other can check.
CloudNala's view is that a rubric line is a specification for a judgement, and it should be written with the same care as any other specification: one behaviour, a plain question, a small scale, an example for each level, a stated gate type, a rule for when it does not apply, and the evidence the grader must cite. A line that is missing these is an adjective, and adjectives cannot be tested.
This is Part 2 of a six-part series. Part 1 argued that the rubric is where policy becomes agent behaviour, so it needs an owner. This part is about writing it well enough to deserve one. CloudNala has already published the anatomy of a useful software specification; a rubric line is the same discipline applied to judgement rather than code.
Kingfisher Contact Solutions is fictional, and its systems and figures are illustrative. It is a Durban contact centre whose AI agent-assist drafts replies and after-call summaries for three fictional client brands: Aloe Cover, Coastlink and Harbour Home.
Why "be helpful" cannot be scored
Kingfisher ran a simple test before rewriting anything. Two experienced quality analysts each scored the same 60 agent-assist drafts against the existing rubric, without seeing each other's scores.
On grammar, they gave the same score almost every time. On "accuracy", they mostly agreed. On "empathy", they agreed little more than chance would predict. One analyst gave high marks to any reply that opened with an apology. The other marked those replies down as formulaic and gave high marks to replies that addressed the customer's specific frustration without apologising at all.
Both were applying a sensible idea of empathy. The rubric line simply did not say which idea it meant. Every vague line in a rubric works like this: it hands the definition to the grader, and different graders hold different definitions. When the grader is a model, the definition is whatever the model's training happens to associate with the word, which is harder still to see.
Part 3 covers how to measure agreement properly. The point here is simpler. If people cannot agree on a line, the fix is to rewrite the line, not to find a more confident grader.
Eight parts of a rubric line that can be scored
A usable rubric line has eight parts. A rubric written in a hurry often has only two or three of them.
1. Criterion. One observable behaviour. "Names the customer's actual problem" is one behaviour. "Empathetic and professional" is at least two, and probably five.
2. Question. The criterion phrased as something a grader can answer by looking at the output. "Does the reply state the customer's problem in terms the customer used?" can be answered. "Is the reply empathetic?" invites an opinion.
3. Scale. As few levels as the decision needs. Two levels (yes or no) or three (no, partly, yes) are easier to apply consistently than five. A five-point scale tends to invite graders to cluster around the middle and argue about the difference between a 3 and a 4.
4. Anchors. For each level, a short description and ideally a real example. Anchors are what turn an adjective into a test. They are also where most of the owner's intent lives, so they deserve the most review.
5. Gate type. Whether the line is must-pass or weighted. This is a decision, not a detail, and the next section explains why.
6. N/A rule. When the line does not apply, and what happens then. "States the next step and when it will happen" does not apply to a call that ended fully resolved. Without an N/A rule, graders either score it as a pass, which inflates the result, or as a fail, which punishes good calls.
7. Evidence. What the grader must quote or point to in order to justify the score. Requiring evidence makes scores checkable, and in CloudNala's view it is one of the most useful controls on a model judge, as Part 3 shows.
8. Aggregation. How this line combines with the others: which lines are weighted and by how much, and what a must-pass failure does to the overall result.
Anchors beat adjectives
Of the eight parts, anchors do the most work. Consider Harbour Home's wish for replies that "sound like us". Harbour Home's brand guide says the voice is "warm, plain and confident". Those are still adjectives.
The anchored version Kingfisher wrote for Harbour Home's layer looks like this:
| Level | Anchor | Example opening |
|---|---|---|
| 0 | Generic contact-centre phrasing | "We apologise for any inconvenience caused." |
| 1 | Plain, but not in the brand's voice | "Sorry your couch arrived damaged. We will look into it." |
| 2 | Plain, specific and confident, as in the brand guide | "Your couch should not have arrived like that. Here is what we will do today." |
Two analysts using these anchors agreed far more often than they had with the word "voice" alone. Just as important, the brand owner could look at the table and say whether it was right. The anchors made the brand owner's intent reviewable, which the adjective never did.
Good anchors come from real outputs, not invented ones. Kingfisher took the anchors for each level from actual drafts in its history, with customer details removed, and the brand owner chose which example belonged at which level.
Must-pass items are not heavily weighted items
The most important structural choice in a rubric is which lines are must-pass. A must-pass line works like a gate: if the output fails it, the output fails, whatever its other scores. A weighted line contributes to a total.
The temptation is to give compliance items a large weight instead. That feels safer than a hard gate, because one strict line will not "unfairly" sink an otherwise good reply. It is exactly backwards. Under a weighted scheme, a fluent, warm, complete reply that omits Aloe Cover's required waiting-period disclosure can still score well overall. The average hides the one failure that matters most.
Kingfisher's rule is now simple. A line is must-pass if its failure would be reported to a client or a regulator, or would require someone to contact the customer again to correct it. Everything else is weighted. The must-pass layer is short, owned by compliance and visible in every report as a separate pass rate, never folded into the quality score.
The same principle runs through the Release on Evidence series, where readiness scores need hard gates the number cannot outvote. A rubric is a small readiness score for every output, and it needs the same protection.
N/A is a decision, and so is evidence
Two more parts deserve a word, because they are the ones most often left implicit.
An N/A rule has to say not only when a line does not apply but also how N/A affects the total. Kingfisher excludes N/A lines from both the numerator and the denominator, so a resolved call is not punished for having no next step, and it records the N/A rate per line. A line that is N/A on half of all calls is a sign it was written for a narrower situation than its owner thought.
An evidence requirement asks the grader to quote the part of the output that justifies the score. For "Never blames the customer or a colleague", the evidence on a fail is the offending sentence. For a pass on "Names the customer's actual problem", it is the sentence that does so. When a model is grading, the quoted evidence lets a reviewer check the judgement in seconds. When the model cannot find a quote to support its score, that is itself a useful signal that the score is unreliable.
One rubric line, rewritten
Here is the original Kingfisher line that started the work, and what replaced it.
The original line asked graders to score "empathetic and professional" from 1 to 5. It had no anchors, no evidence requirement and no N/A rule, and it averaged together several things that matter very differently.
The rewrite splits it into three lines. The first, "Never blames the customer or a colleague", is must-pass: blame in a customer reply is something Kingfisher's clients have told it they will escalate, and averaging it away is not acceptable. The second, "Names the customer's actual problem", is weighted from 0 to 2 with anchors. It captures the part of empathy that the analysts who disagreed both recognised as real. The third, "Says what happens next, and when", is weighted from 0 to 2, and is N/A when the call ended fully resolved.
Notice what disappeared: the word "professional". When the owners were asked what it meant, the answers were "does not blame anyone", "no slang" and "correct spelling". The first became a must-pass line, slang was rare enough to be handled by a spot check, and spelling was already covered elsewhere. A word that sounded essential turned out to be three smaller things, one of them important.
Start this week
- Pick your three most important rubric lines and check each one against the eight parts. Note which parts are missing.
- Run a two-grader test on one vague line: two people score the same 30 to 60 outputs independently. If they disagree often, rewrite the line before doing anything else with it.
- Write anchors from real outputs. For each level, find a real example in your history, remove personal information, and ask the line's owner to confirm it belongs at that level.
- Move must-pass items out of the average. Report them as a separate pass rate.
- Add an evidence requirement to every line, starting with the must-pass ones.
Next: Part 3, Calibrate the judge before you trust the score. All parts are on the series page.
How CloudNala can help
CloudNala helps teams rewrite evaluation rubrics so they can be scored consistently. We work with the people who own each decision to turn goals into anchored lines, separate must-pass conditions from weighted criteria, write N/A and evidence rules, and test each line with real graders on your own outputs before it reaches a model judge or a client report.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za