Many organisations that run an AI agent in production now have some form of evaluation. A set of test cases, a scoring sheet, perhaps a model that grades the agent's output against a list of criteria. The scores go into a weekly report, and the report says the agent is doing well or badly.
What the report rarely says is who decided what "well" means. That decision lives in the rubric: the list of criteria, the scale for each one and the rules for combining them. And the rubric does something the policy documents never do. It sits inside the feedback loop. Engineers change prompts, swap models and adjust tools until the scores improve. Over weeks and months, the agent becomes whatever the rubric rewards.
CloudNala's view is that a rubric is not a neutral measuring instrument that sits outside the agent. It is the most concrete statement of policy the agent will ever see, and whoever writes it is setting the agent's behaviour. Rubric authorship is a governance decision with a named owner, not a configuration task left to whoever set up the evaluation tool.
This six-part series is about that decision and the engineering that makes it hold. This first part covers who should own a rubric and why. Part 2 shows how to write rubric lines that can actually be scored. Part 3 covers calibrating a model judge, Part 4 versioning, Part 5 applies the same discipline to coding-agent harness experiments, and Part 6 sets out a minimal evaluation lab you can build in a week.
Kingfisher Contact Solutions is fictional, and its systems and figures are illustrative. Kingfisher is a Durban contact centre with about 600 agents, serving three client brands: Aloe Cover, a funeral and health-cover insurer; Coastlink, a fibre internet provider; and Harbour Home, a furniture retailer. Eight months ago it introduced an AI agent-assist that drafts replies for human agents and writes after-call summaries. Its engineering team of 12 also runs a coding-agent pilot, which appears in Parts 5 and 6.
Where policy becomes behaviour
Kingfisher's agent-assist scored 4.3 out of 5 for "quality" in its June quarterly report. All three client brands were unhappy, and for three different reasons.
Aloe Cover's compliance team found draft replies that skipped a required disclosure about waiting periods. Coastlink's operations lead complained that replies were long and warm but rarely solved anything on the first contact. Harbour Home's customer experience manager said the replies sounded like every other company's, which was the opposite of the brand's voice.
The rubric explained all three. It had been written in the first week of the rollout by the tool vendor's onboarding consultant, adapted by a Kingfisher team lead and never reviewed again. It had five criteria, each scored from 1 to 5 and averaged: "empathetic and professional", accuracy, completeness, clarity and grammar. The empathy line and completeness together made up 40% of the score. Required disclosures did not appear at all, except as one part of "completeness".
Over four months, the engineers had done exactly what engineers should do with a measure. They improved it. Prompts gained instructions to acknowledge feelings and to cover every possible next step. Replies got longer and warmer, and the scores rose. Nobody had decided that Kingfisher's agents should write long, warm, generic replies. The rubric had decided it for them.
This is the mechanism to understand. Policy documents, brand guidelines and service standards describe intent. They do not change the agent's behaviour unless something translates them into a signal the delivery loop responds to. In most AI teams, that signal is the evaluation score. So the rubric is where policy becomes behaviour, and any gap between the two is filled by whatever the rubric happens to reward.
The same idea is familiar from management. A widely quoted line, usually attributed to the economist Charles Goodhart and popularised in Marilyn Strathern's 1997 wording, says that "when a measure becomes a target, it ceases to be a good measure". With AI agents the effect is faster, because the optimisation is cheap. Changing a prompt and re-running a test suite takes minutes.
Four people who think they own the rubric
When Kingfisher's head of operations asked who owned the rubric, four people said they did, and each would have written a different one.
The client brand owner wants the agent to sound like the brand and to protect the brand's customers. Harbour Home's manager would weight voice and tone heavily. Aloe Cover's would make disclosures non-negotiable.
The contact centre operations manager is paid on handling time, first-contact resolution and quality audits. They would reward replies that resolve the issue quickly and penalise anything that causes a repeat call.
Compliance cares about a small number of things that must always or never happen: required disclosures, no promises the agent cannot authorise, no personal information about the wrong person. To compliance, these are not criteria to be averaged. They are conditions.
The engineer or vendor who set up the evaluation usually writes the first version, because someone has to. They reach for generic criteria that any agent could be scored on, because they do not yet know the business well.
None of these people is wrong. The problem is that the rubric is the place where their priorities are traded off against each other, and at Kingfisher the trade-off was made by default, by the person with the least context.
Separate the decisions, then name the owners
"Who owns the rubric?" is really five questions, and they have different answers.
What good means is a business decision. For a brand-facing agent, the brand owner and the operations manager decide it together, because both carry the consequences. The vendor's default criteria are a starting point at most.
The wording of each line is a craft job for whoever runs quality assurance. Turning "the reply should sound like Harbour Home" into something a grader can score consistently is skilled work, and Part 2 covers how to do it.
The must-pass floor belongs to compliance. These items should not be weighted into an average at all, because an average lets a warm, fluent reply outscore a missing legal disclosure.
Approving a change is the decision that most often goes wrong. If the team whose scores improve when the rubric changes is also the team that approves the change, the rubric will drift towards whatever the agent already does well. Kingfisher now has a named rubric owner per brand who signs off changes, and that owner is never the engineer tuning the prompts.
Running the scoring is an engineering job: the runner, the judge, the storage of results. Engineering owns how scores are produced, not what they mean.
This split mirrors a principle the AI Before the Rulebook series makes about decisions generally: AI may assist the decision, but it must not inherit authority. The same applies to the people around the agent. Running the evaluation does not confer authority over what the evaluation measures.
One rubric, or one per brand?
Kingfisher's original rubric scored all three brands the same way. That was part of the problem: Coastlink's idea of a good reply and Harbour Home's are different, and averaging them produced a reply that suited neither.
The answer Kingfisher settled on is a layered rubric. At the bottom is a shared must-pass floor that applies to every brand: no blaming the customer, no promises the agent cannot authorise, no information about anyone other than the customer on the call, and every disclosure the brand's rules require. Above that is a short Kingfisher service standard that all brands share, such as stating the next step and when it will happen. On top, each brand owns a small set of weighted lines for its own voice and priorities.
The layers make the trade-offs explicit. A brand can weight its own lines however it likes, but it cannot weight its way past the floor. And when a brand changes its layer, the change affects only its own agents' scores, so the other two brands do not inherit a shift in behaviour they never asked for.
The rubric is also a contract
For an outsourcer like Kingfisher, there is a commercial point too. Each client contract says something about quality. Until the rubric review, those clauses referred to "quality scores" without saying how quality was defined. In practice, the definition was a file one team lead had edited in the first week of the rollout.
Kingfisher now attaches each brand's rubric layer to the service schedule, with its version number. A change to the layer is a change to what the client is buying, and the client signs it. That has a second benefit: when a client asks why a score moved, there is a written answer, not a debate about what "quality" meant last quarter.
The same logic applies inside a single organisation. If a business unit's targets or a regulator's expectations are being enforced through an AI agent, the rubric is the operative definition. It deserves the same version control and sign-off as any other policy instrument.
Signs that your rubric has no owner
Most organisations can check their own position quickly. The rubric has no owner if any of these are true:
- nobody can say who last changed it, or why;
- the criteria are generic enough to score any agent in any organisation;
- must-pass items, if they exist, are weighted into an average;
- the person who approves changes is the person whose results improve;
- different teams or clients are scored by one rubric that none of them chose;
- the scores have gone up steadily while complaints have not gone down.
That last sign is the most telling. Rising scores and flat or rising complaints usually mean the agent has learned the rubric rather than the job. The Agent Evidence Layer series covers how to connect evaluation results to real release decisions; this series is about making sure the thing being measured is the thing you meant.
Start this week
- Find your rubric and read it as policy. Ask of each line: if the agent maximised this, would we be happy? Mark every line where the answer is "not entirely".
- Name an owner for each of the five decisions in the diagram above. Write the names down. Where the honest answer is "the vendor" or "nobody", that is your first gap.
- Pull out the must-pass items. List the things that must always or never happen, take them out of any average, and treat a failure on any one as a failure of the whole reply.
- Check the trend. Put your evaluation scores next to complaints, escalations or rework for the same period. If they disagree, believe the complaints and review the rubric before you touch the agent.
Next: Part 2, A rubric is a specification for judgement. All parts are on the series page.
How CloudNala can help
CloudNala runs evaluation design sessions for organisations operating AI agents. We review your current rubric as a policy document, map who should own each decision about it, separate must-pass conditions from weighted criteria, and set up a change process that keeps the people tuning the agent from quietly redefining success. Where clients or regulators depend on your quality scores, we help you make the rubric the written, versioned definition behind them.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za