Kingfisher Contact Solutions' engineering team of 12 has run a coding-agent pilot for four months. For most of that time, its retrospectives have featured the same argument.
One senior engineer believes the agent works best with a long, detailed instructions file covering the codebase's history, conventions and pitfalls. Another believes the long file confuses the agent and that a short list of hard rules works better. A third wants to give the agent read access to the test database. The team lead wants a mandatory self-review step before the agent opens a pull request. Everyone has examples. Nobody has evidence.
CloudNala's view is that arguments about a coding-agent harness are empirical questions, and most teams already have what they need to answer them: a backlog of real tasks with known-good outcomes, a rubric for what an acceptable change looks like, and the patience to run each variant more than once. Run the experiment, decide in advance what result would change your mind, and stop debating.
This is Part 5 of a six-part series about evaluation as governance. Parts 1 to 4 dealt with rubrics for an agent's output. This part applies the same discipline to the harness around a coding agent. CloudNala has explained what a harness is in Guides, guards, checks and sensors and Harness engineering has grown up; this part is about how to tell whether a change to one actually helps.
Kingfisher Contact Solutions is fictional, and its systems, tasks and figures are illustrative. Its engineering team maintains the agent-assist platform and the integrations with its three client brands' systems.
Why anecdotes do not settle it
Each engineer's examples were real. The problem is what they can prove.
The same agent on the same task does not always produce the same result. Language models sample their outputs, and a coding agent makes dozens of decisions on a task. Run it twice and it may take different paths, one of which works. This variability is well documented. In the 2021 paper that introduced OpenAI's Codex, the authors reported that repeated sampling was "a surprisingly effective strategy", with one model solving 28.8% of their benchmark problems with a single sample and 70.2% when 100 samples were drawn per problem (checked on 31 October 2026). The flip side for anyone evaluating a harness is that a single run tells you little. A variant that "worked on my ticket" may simply have been lucky.
The tasks were different. The long-file advocate's examples were refactors in an old module, where history matters. The short-rules advocate's examples were new API endpoints, where it does not. Both may be right about their own tasks.
Everything else changed too. Over four months, the model version changed twice, the codebase grew and the engineers got better at writing tickets. Comparing "before" and "after" a harness change mixes all of that together.
An experiment controls for these things. It does not need to be elaborate.
One variable, everything else held still
Kingfisher's first experiment tested the question that had caused the most argument: a long instructions file against a short, rule-based one.
The hypothesis. Written in one sentence before anything ran: "A short instructions file of hard rules will produce more changes that are accepted without rework than the current long file, across a representative set of our tasks."
The task set. Twenty tickets the team had already closed in the last three months, chosen to represent the real mix: six bug fixes, five small features, four refactors, three integration changes and two test-only tasks. Each had a merged human solution, so the team knew what a good outcome looked like. Tasks that relied on information outside the repository, such as a phone conversation with a client, were excluded, because no harness could have succeeded on them.
The environment. For each task, a snapshot of the repository at the commit before the original fix. Both variants start from the same snapshot.
Held constant. The model and its version, the tools available to the agent, the time limit per run, the automated checks and the review rubric. Only the instructions file differs between variant A (the existing long file) and variant B (the short one).
Repeated runs. Each variant runs each task three times. Three is a small number, chosen for cost; it is enough to see when a variant is consistently better or worse on a task, and to stop one lucky run from deciding the result.
Measure what a delivery lead would recognise
The agent's own report that it finished the task is not a measure. Kingfisher scored every run on four things.
Accepted change. Would the change be merged as it stands? This combines deterministic checks (the build passes, the existing tests pass, the linter is clean, the tests from the original fix pass) with a short review rubric written in the style of Part 2. The rubric has one must-pass line ("does not change behaviour outside the ticket's scope") and three weighted lines covering clarity, test coverage and consistency with the codebase's conventions.
Reviewer minutes. How long a reviewer took to reach a decision. A change that is technically correct but takes 40 minutes to understand is costly.
Cost and elapsed time. The model usage cost of the run, converted to rand, and wall-clock time. These matter less than acceptance but break ties.
Rework within two weeks. In a live trial, whether the change needed fixing after merge. In this offline experiment, Kingfisher approximated it by having a second reviewer look for defects the first had missed.
These line up with the measure argued for in the Economics of AI Delivery series: cost per accepted outcome, not cost per prompt. A harness that makes the agent cheaper per run but produces fewer acceptable changes is not cheaper.
Reviewers did not know which variant produced the change they were scoring. Blind review is easy to set up, by stripping the branch name, and it removes one obvious source of bias in a team that has been arguing for months.
Write the decision rule first
Before any runs started, the team agreed what result would change their minds. The rule was: adopt variant B if it produces at least ten more accepted changes across the 60 runs (20 tasks, three runs each) and does not increase the number of runs that break an existing test. Otherwise, keep A.
Writing the rule first matters more than any statistic. Without it, every result can be argued away: the long-file advocate could say ten more accepted changes is small; the short-rules advocate could say five more is meaningful. With it, the argument happens once, before anyone knows which way the evidence falls.
Reading the result
Variant B produced 38 accepted changes against A's 27, broke tests in three runs against A's five, and reduced median reviewer time from 22 minutes to 14. Cost per accepted change fell slightly. The pre-agreed rule was met, and the team adopted the short file.
The more useful part of the result was the detail. B did worse than A on four of the 20 tasks, and three of those four were refactors in the oldest module, exactly the cases the long-file advocate had cited. The history in the long file mattered for that module. So the team did not simply delete it. It moved the module-specific history into a short file that sits inside that module's directory, where the agent reads it only when working there, and added more refactors from that module to the task set so the next experiment could test the change.
The other result worth noting is what the experiment could not show. Twenty tasks and three runs each is enough to see a large difference, not a small one. If B had produced 30 accepted changes against A's 27, the honest conclusion would have been "no clear difference", and the team would have kept A on the grounds that changing it was not worth the disruption.
Make it a habit, not a project
The first experiment took Kingfisher about four days of one engineer's time, plus the reviewers' hours, most of it spent choosing tasks and building the snapshots. The second took a day, because the task set and runner already existed.
The team now handles harness proposals the way it handles other changes. Anyone can propose a change, but it comes with a one-sentence hypothesis and a decision rule, and it runs against the task set before it is adopted. The experiments so far have tested the instructions file, read access to a seeded test database (adopted for integration tasks only) and the mandatory self-review step (no clear difference in acceptance, a clear increase in cost, not adopted).
Each experiment produces a one-page write-up: the hypothesis, the variable, the task set version, the decision rule, the results table, the decision and any tasks added to the set as a result. Those write-ups have replaced the retrospective arguments. When someone raises an old question, the answer is a link.
Start this week
- Write down the harness question your team argues about most, as a one-sentence hypothesis.
- Pick 10 to 20 closed tasks from your own backlog that have merged human solutions and do not depend on information outside the repository. Cover the real mix of work.
- Write the review rubric and the decision rule before running anything.
- Run each variant on each task at least three times from the same repository snapshot, with reviewers blind to the variant.
- Publish a one-page write-up, including the tasks the winning variant did worse on, and add more tasks like them to the set.
Next: Part 6, An evaluation lab you can build in a week. All parts are on the series page.
How CloudNala can help
CloudNala helps engineering teams turn coding-agent debates into experiments. We help you select a representative task set from your own backlog, build repeatable snapshots, write the review rubric and decision rule, run the comparison and interpret the result honestly, including when the answer is "no clear difference". The aim is a harness your team can defend with evidence, and a habit of testing changes before adopting them.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Evaluation Design Session or write to us at consult@cloudnala.co.za