Whoever Writes the Rubric Shapes the Agent
Every team running an AI agent tunes it until the evaluation scores improve, so the rubric behind those scores ends up steering the agent. This six-part series treats rubric authorship as governance: who should own each decision about a rubric, how to write lines that can be scored consistently, how to calibrate people and then a model judge, how to version a rubric so a change is never mistaken for a change in the agent, how to settle coding-agent harness arguments with a small controlled experiment, and a minimal evaluation lab you can build in a week. It follows Kingfisher Contact Solutions, a fictional Durban contact centre running an AI agent-assist for three fictional client brands and a coding-agent pilot; its systems and figures are illustrative. Research claims were checked on 31 October 2026.
6 articles
- Part 1Whoever writes the rubric shapes the agentAn evaluation rubric looks like a measuring instrument. In practice it is the most concrete statement of policy an AI agent will ever meet, because every team tunes its prompts and models until the scores go up. Part 1 shows why rubric authorship is a governance decision, who should own each part of it, and how to tell when nobody does.9 min read
- Part 2A rubric is a specification for judgement"Be empathetic and professional, 1 to 5" cannot be scored consistently by people or by a model. Part 2 sets out the eight parts of a rubric line that can be scored, why anchors matter more than adjectives, why must-pass items must never be averaged, and rewrites one vague line into three that work.9 min read
- Part 3Calibrate the judge before you trust the scoreUsing a language model to grade another model's output is now common, and the scores look precise. Part 3 shows how to earn that precision: calibrate your human graders first, measure the model judge against them, design around its known biases, choose the right checker for each criterion and keep an audit sample running after launch.8 min read
- Part 4Version the rubric like codeA rubric that changes without a version number quietly rewrites your history: last quarter's scores stop meaning what they meant. Part 4 covers rubric change control, version types, re-scoring a golden set before every release, telling score drift from behaviour drift, and keeping an agent from learning the rubric instead of the job.8 min read
- Part 5Stop debating harnesses. Run the experiment.Teams argue for weeks about coding-agent instructions files, tools and review gates, usually from anecdotes. Part 5 shows how to settle those arguments with a small controlled experiment: a fixed task set from your own backlog, one variable at a time, repeated runs, measures a delivery lead would recognise, and a decision rule written before the results.8 min read
- Part 6An evaluation lab you can build in a weekEvaluation does not need a platform to start. Part 6 sets out a minimal evaluation lab with seven parts, a day-by-day plan for building it in five working days, the weekly review that keeps it honest, and what to deliberately leave for later. It closes the series with CloudNala's view on who should own the definition of good.8 min read
Working through any of this in your own organisation?
These articles come out of delivery work rather than theory. If the problems in this series look familiar, a short scoping conversation is usually the quickest way to work out what is worth doing first.
Talk to CloudNala