Swartland Mutual Bank's design document for its debt-review inbox contained the sentence most AI designs contain: "Items below the confidence threshold are reviewed by a person." Swartland is the fictional South African bank followed through this series, and the sentence was true. Every low-confidence field went to a reviewer.
Two months after launch, the bank's internal audit team asked two questions that nobody could answer. How accurate were the fields that had been accepted without review? And how accurate were the reviewers? The design had placed people in the loop. It had not measured the loop.
This part is about that gap. Other CloudNala articles cover where a person should intervene (Human-in-the-loop is not a button), how review feedback improves a system (Human-in-the-loop is not a bottleneck), why verification must be independent and leave a trace (The human in the loop must show their work) and why a review step is not the same as administrative justice (Human-in-the-loop is not administrative justice). Here the subject is narrower: how to measure a review process so that you know what it is catching and what it is missing.
Review is a control only when it produces evidence
A review step does two jobs. It corrects individual items, and it is a control the organisation relies on for its overall accuracy. The first job happens automatically. The second happens only if the review produces data that someone analyses.
For every reviewed item, Swartland began to record:
- the value the system proposed and its confidence;
- the value the reviewer accepted, and whether it changed;
- if it changed, the error class (see below);
- the time the reviewer spent; and
- the reviewer, the model version and the threshold in force at the time.
None of this needs a new platform. It is a few fields in the case record described in Every automated recommendation needs a case file. Without them, review is effort. With them, it is evidence.
Set thresholds from evidence, then keep checking them
A confidence score is only useful if it means what it says. A field the model scores at 0.95 should be right about 95% of the time, and many models are not that well calibrated out of the box.
Swartland set its thresholds with a labelled sample of 2,000 live fields per document type, collected during a four-week shadow period in which the system ran but people still keyed everything. The method was straightforward.
- Group the sample into confidence bands: 0.99 and over, 0.97 to 0.99, 0.95 to 0.97 and so on down.
- For each band, measure the observed error rate against the keyed values, after resolving disagreements.
- Agree with the process owner the highest error rate the business will accept for auto-accepted fields of that type.
- Set the threshold at the lowest band whose observed error rate stays below that tolerance.
The tolerance was a business decision, not a technical one, and it differed by field. A misread date on a consumer letter is a nuisance. A wrong account number on a Form 17.1 notice is a serious error. The figures in this article are illustrative.
Thresholds were then reviewed monthly against fresh evidence, because calibration drifts. A model update, a new counsellor template or a seasonal shift in document mix can move it. Thresholds, evals and human review makes the related point that thresholds belong to the action they permit; here the emphasis is on evidence that the threshold still holds.
Sample the items nobody reviews
The most important measurement is also the most often skipped. Once a field is above its threshold, nobody looks at it. If the model is wrong there, nobody finds out until a customer, a counsellor or an auditor does.
Swartland draws a random sample from the auto-accepted band every month, 300 items stratified across document types, and has them checked by a reviewer who does not see the model's value first. That last condition matters. A reviewer shown a value tends to confirm it.
The sample size was chosen with a simple statistical rule. If you check n items at random and find no errors, the "rule of three" puts the upper limit of a 95% confidence interval for the true error rate at about 3/n (Hanley and Lippman-Hand, 1983). So 300 clean samples suggest the error rate is below about 1%. They cannot show it is zero, and 300 samples cannot detect an error rate of one in ten thousand. If your tolerance is tighter, you need a much larger sample or a different control.
Two refinements followed from early results:
- Oversample where errors cost most. Notices and proposals, where a wrong account match is serious, were sampled at three times the rate of general correspondence.
- Oversample new sources. When a debt-counselling firm's documents appeared for the first time, its items were sampled more heavily for the first month.
An error taxonomy turns corrections into fixes
A correction that is not classified teaches nothing. Swartland's reviewers tag every change with one of seven classes, each with a severity and an owner.
The taxonomy changes the conversation in the monthly review. "The error rate is 1.8%" invites an argument about whether that is good. "Most of this month's errors are poor source documents from two counsellor firms, and the intake team owns the fix" invites action. It also stops every problem being sent to the model team. In Swartland's first quarter, fewer than half the corrections were model errors in the narrow sense. The rest were intake quality, missing rules and reviewer slips.
Two rules keep the taxonomy useful. It stays small, because reviewers will not use twenty classes accurately. And a class with a rising trend goes to its owner with examples attached, not just a count.
Measure the reviewers too
Reviewers are part of the system, and their accuracy is not guaranteed by their presence. Swartland measures three things.
Agreement. Five per cent of reviewed items are reviewed again, blind, by a second reviewer. Where the two disagree, a senior adjudicator decides. Agreement is tracked per field type using Cohen's kappa, which corrects for agreement expected by chance; Calibrate the judge before you trust the score explains the statistic and why raw percentage agreement misleads. Low agreement on a field usually means the guidance is unclear, not that a reviewer is careless.
Accuracy against known answers. A small number of items with known correct values are seeded into review queues each week. Reviewers do not know which ones. This measures whether reviewers catch errors, rather than whether they agree with each other.
Signs of rubber-stamping. If a reviewer's change rate falls close to zero, or their time per item drops sharply, the queue may have taught them that the system is always right. That is a known risk, discussed in Human-in-the-loop is not a button. The seeded items give an early warning.
None of this is about ranking people. Reviewer measures are reported by team and by field, and the response to a weak result is better guidance, better tooling or a rebalanced queue.
Close the loop: escalation and correction
Measurement is only worthwhile if it changes something. Swartland's loop has four paths.
- Disagreement escalates. When two reviewers disagree, or a reviewer is unsure, the item goes to an adjudicator, and the decision becomes guidance for everyone.
- Corrections become labels. Every adjudicated correction joins the evaluation set used to test the next model or template change, which is how the discipline in Evals that gate real work is kept supplied.
- Patterns become rules or upstream fixes. A repeated rule gap becomes a written rule with an owner. A repeated quality problem goes back to intake, as described in part 3.
- Evidence moves thresholds. If the sampled error rate in the auto-accept band rises above tolerance, the threshold rises until the cause is fixed. If it stays well below for several months, the owner may lower the threshold, with the evidence recorded.
A review measurement checklist
| Question | Evidence that answers it |
|---|---|
| What is recorded for every reviewed item? | Proposed value, final value, error class, time, reviewer, model version, threshold |
| How was each threshold set? | Observed error rate by confidence band on a labelled live sample, against an agreed tolerance |
| How often are thresholds re-checked? | A monthly review with fresh samples, and after every model or template change |
| What is the error rate in the auto-accepted band? | A random, blind, stratified monthly sample, with its confidence interval stated |
| Why do errors happen? | A small error taxonomy with severity and owner per class |
| How good are the reviewers? | Blind second review with kappa, and seeded known-answer items |
| What changes as a result? | Escalation, new labels, rules or intake fixes, and threshold changes, each recorded |
Key points
- A person in the loop corrects items. Measurement of the loop is what makes it a control.
- Set thresholds from observed error rates by confidence band, against a tolerance the business owns, and re-check them monthly.
- Sample the auto-accepted band at random and blind. The rule of three tells you what a clean sample can and cannot prove.
- Classify every correction. An error taxonomy with owners turns corrections into fixes, many of them outside the model.
- Measure reviewers with blind second review and seeded known answers, and watch for rubber-stamping.
- Close the loop: escalate disagreements, turn corrections into labels, fix causes upstream and move thresholds on evidence.
This is part 4 of AI That Changes the Work, Not Just the Interface. Previous: The AI is only one step in the process. Next: Measure the queue, not the demo.
How CloudNala can help
CloudNala helps operations, risk and audit teams turn human review into a measured control. We design the review record, set thresholds from a shadow-period sample, specify the sampling plan for auto-accepted work, build the error taxonomy with clear owners, and set up reviewer agreement and seeded-item checks, so that you can say with evidence how accurate the whole process is, not only the part a person saw.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Operational AI Review or write to us at consult@cloudnala.co.za