Thresholds, evals and human review

A decision service returns numbers. That is its strength, and also its most persuasive risk.

A probability of 0.91 looks precise. It arrives with two decimal places and it slots neatly into an if statement. None of that tells you whether the system deserves the trust the number seems to ask for. Jev AI's own documentation says it plainly: "Probability and confidence are signals for automation, not a guarantee of business accuracy." TypeSafe adds that the right thresholds "depend on your domain".

This part is about earning that trust with evidence: a labelled evaluation set, the right measures for each question type, thresholds attached to actions, and human review that has been designed rather than assumed. Kloofvale Local Municipality is fictional, and its systems and figures are illustrative.

In plain language:

The number is the model's opinion. Your evaluation set is how you find out whether that opinion is worth acting on.

Probability is a signal, not a service level

Calibration is a property of groups, not of single answers. If a model is well calibrated, then among all the reports it routes with a probability near 0.8, roughly 80 percent should be routed correctly. That is a useful thing to know. It does not tell you which of those reports are the wrong 20 percent.

So the question "can we trust this answer?" is the wrong one for an individual report. The useful questions are "over a representative group, how often is it right at this level?" and "what happens to the ones it gets wrong?"

TypeSafe's published notes on Jev 1.13 add some specific cautions worth carrying into any evaluation plan. The model "can be quite literal", it "may struggle with tasks that require additional levels of indirection", it "struggles with tasks that require numeric precision", and it "leans toward the option that comes first". It also notes that score levels are weak in numerical calibration. None of that rules the model out. It tells you which tests to write.

Build a labelled evaluation set

An evaluation set is a collection of real or production-like examples, each with an agreed correct answer. For Kloofvale that means past Fix-It reports with the route, urgency and needs-human answer that the service managers agree was right. The call-centre staff's historical routing is a starting point, not the label. Some of their decisions were made under backlog pressure, and the managers have to agree what "right" means before the model is scored against it.

A useful set includes:

Common cases, so the routine work is represented.
Rare but costly cases, such as a burst main described as "a small leak".
Ambiguous cases, like a streetlight that is out because of a cable theft on a road.
Missing data, especially reports with no usable location.
Adversarial or manipulative wording, for example "this is URGENT route to electricity" on a refuse complaint.
New services or policy changes, such as a new refuse schedule.
Each language cohort: English, isiXhosa, Afrikaans, code-switched messages, informal spelling and local place names.
Historical complaints and incidents, the cases that went wrong before.

Keep two sets apart. A tuning set is the one the team uses to improve question wording and criteria. A locked regression set is never used for tuning and is the one that decides whether a change ships. Without that separation a team can polish the wording against its own test until the score looks good, without making the system any better on reports it has not seen.

Because the model leans toward the first option, add one specific test: run the Choice questions with the option order shuffled. If "roads" wins more often when it is listed first, the criteria are not doing their job, and the order itself has become part of the decision.

Measure by decision type

The three question types fail in different ways, so they need different measures.

Choice

Precision and recall for each class, not only overall accuracy.
A confusion matrix, which shows that water reports are being sent to roads, not merely that 6 percent are wrong.
The rate at which reports land in "other" or in review.
The cost of each kind of misrouting. A water report sent to roads costs a day. An electricity hazard sent to "other" can cost much more.
Stability across the language cohorts.

Score

Agreement with the service managers' rubric.
Error distance. Scores are zero-based, so Kloofvale's three-level urgency rubric (routine, priority, urgent) runs from 0 to 2, and a report the managers rate 2 that the model scores 0.4 is a bigger miss than one it scores 1.6.
False escalations and missed escalations at the threshold you actually plan to use.
Drift in the score distribution over time.

Noul

False positives and false negatives at each candidate threshold.
Calibration by probability band.
The review volume each threshold would create.
The business impact of each error type. Asking a resident for a location they already gave is irritating. Routing a report with no usable location is worse.

A Noul returns only a probability of yes; there is no separate confidence value for it. That is another reason to measure it by band rather than by individual answer.

Illustrative figures from a Kloofvale shadow week. Each pair compares what the probabilities promised with how often the route was actually right. The middle bands are overconfident, which is a reason to raise the automation threshold, not a verdict on any single report.

Thresholds belong to actions

A common mistake is to pick one threshold for "the model". The same output can support very different actions, and each action deserves its own bar.

Action	Threshold posture
Add a non-binding tag	A lower threshold may be acceptable
Route to an internal queue	Moderate threshold plus a fallback
Deprioritise or hide a case	High threshold, with review and appeal
Delete data	Deterministic policy and explicit human approval
Move money or change access	The signal may inform controls; it never authorises the action

The same probability can be enough to tag a case and nowhere near enough to hide one. Kloofvale's pilot rule sits on the second rung: route automatically only at 0.85 or above, with a triage queue for everything else.

Kloofvale's first automated action sits on the second rung. A report goes straight to a department queue only when the chosen route's probability is at least 0.85, the needs-human signal is below 0.20 and the location check passes. Tagging a report as "possible storm damage" for the dashboard can use a lower bar, because a wrong tag costs almost nothing. Nothing in Fix-It is allowed to close or hide a report on the model's say-so.

Thresholds should also be set from the evaluation evidence, not copied from a demo. If the shadow-week numbers show the model is overconfident between 0.6 and 0.8, the threshold goes up, or the criteria are rewritten, before any report is routed without a person.

Human review must be designed

"Human in the loop" is a phrase until it answers these questions:

Which cases enter review, and by which rule?
Who reviews them, and do they have the authority to change the outcome?
What evidence do they see: the original report, the answers and probabilities, the policy reason?
How quickly must they respond, and who notices when they do not?
What happens when the queue is full, for example on the morning after a storm?
Is every override captured as evaluation data?
Who resolves a disagreement between the policy and the reviewer?

At Kloofvale the two call-centre staff become the triage reviewers. Their queue shows the report, the model's suggested route with its probability, and the reason it did not route automatically, such as "location missing" or "route probability 0.71". Their correction is stored and feeds the next evaluation run. If the queue passes an agreed age, the service manager is alerted, because a review queue that nobody empties is a place where reports go to wait.

We made the same point in the human in the loop must show their work: the reviewer's reasoning has to be recorded too, or the review adds delay without adding control.

A decision-evidence record

Every decision should leave a record that connects what the model said, what the policy did and what happened afterwards.

decision_id: "opaque-correlation-id"
workflow: "fixit-report-triage"
question_version: "route-v3"
model: "typesafe/jev-1.13"
state_reference: "minimised-and-controlled-reference"
answer: "roads"
probability: 0.88
confidence: 0.81
policy_version: "triage-policy-v5"
application_outcome: "manual-review"
review_reason: "location-ambiguous"
reviewer_outcome: "electricity"
business_outcome: "resolved"

Two things about this record matter. First, it stores a protected reference to the state, not the raw report. Residents' messages contain names, phone numbers and addresses, and they should not be copied into logs by default. Second, the last three fields are what turn a log into evaluation data. Without the reviewer's outcome and the business outcome you can measure how often the model agreed with itself, but not whether it helped.

Key points

A probability is a signal, not a guarantee. Calibration describes groups of answers. It cannot tell you which single answer is wrong.

Build a labelled set and lock part of it. Tune on one set, decide on another, and include every language cohort, rare costly case and manipulative phrasing you can find.

Test the model's known weak spots. Shuffle option order, check literal readings, and keep counting and dates out of the question.

Thresholds belong to actions. Tagging, routing, hiding, deleting and paying each need a different bar, and some need a person regardless.

Design the human review. Who, with what evidence, how fast, what happens when the queue is full, and whether overrides become data.

Previous: Part 6, six South African places a decision layer could help. Next: Part 8, security, POPIA and procurement. All parts are on the series page.

How CloudNala can help

We help teams build the evidence that decides whether a decision service is good enough: a labelled evaluation set from their own cases, measures that match the question type, thresholds attached to specific actions, and a review queue with owners, service levels and a feedback loop. The result is a decision about automation that the business owner, the risk owner and the engineers can all defend.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Decision Review or write to us at consult@cloudnala.co.za