The steering committee at Swartland Mutual Bank, the fictional South African bank in this series, had seen the vendor demonstration. A debt counsellor's notice was dropped into a folder and, four seconds later, appeared in the case system with every field filled in. The committee wanted that number in the monthly report.
The operations head wanted different numbers. How old was the oldest item in the inbox? How many cases were completed correctly the first time? What did each completed case cost, including the people who reviewed the exceptions? None of those appeared in the demonstration, and all of them decided whether customers in debt review were being served better.
This final part is about choosing measures that tell you whether the work changed. It draws on the earlier parts: the exception queue from part 2, the moving bottleneck from part 3 and the review evidence from part 4.
What the demo measures, and what the customer feels
A demonstration measures the system at its best: one clean document, one fast path, no queue. That is a legitimate thing to show, and a poor thing to manage by. The customer never experiences the four seconds. They experience the total time from the moment their debt counsellor sent the notice until the bank acted on it, and whether the bank got it right.
Each measure on the left of that diagram is real and easy to collect. None of them answers the operational question. Each measure on the right is harder to collect, and answers it. The same argument was made for contact centres in Measure avoided demand, not conversations: a count of activity can rise while the outcome gets worse. This part applies it to the back office.
Eight measures of the queue
Swartland's monthly operations review settled on eight measures, each reported per document type and per week.
- Arrivals and backlog. How many items arrived, and how many were open at the end of the week. Without arrivals, a falling backlog might only mean a quiet week.
- Age: 90th percentile and oldest item. How long the slowest tenth of items had waited, and the age of the single oldest one. More on this below.
- End-to-end cycle time. From arrival to decision, and separately from arrival to case closed, including communication. Part 3 showed why indexing time alone misleads.
- First-time-right rate. The share of cases completed without any correction, rework or re-contact.
- Rework rate. The share of cases reopened, re-keyed or corrected after being marked complete, and why.
- Exceptions by reason. How many items left the automated path, and the reason code for each, from part 2.
- Outcome quality. The sampled error rate on auto-accepted and reviewed items, from part 4, and complaints and escalations that trace back to the process.
- Cost per completed case. Everything it took to complete a case correctly, divided by the number of cases completed correctly. See below.
None of the eight counts model calls, prompts or documents "touched". Those are useful for engineering and cost control, as Four AI engineering metrics teams should measure every week describes, but they are inputs, not outcomes.
Ageing beats averages
The first month's report showed an average time to index of under four hours. It looked like a finished job. The 90th percentile was nearly two days, and the oldest item was 19 days old.
Averages hide the long tail, and in an operations queue the long tail is where the harm sits. The items that wait longest are usually the hard ones: the poor scan, the unfamiliar court-order layout, the notice that matches two customers. They are also the ones most likely to matter, because they are the cases where something is already unusual.
Swartland changed three things as a result:
- The report leads with the 90th percentile and the oldest item, not the mean.
- Every item over five working days old appears by name on a list for the team lead, with its reason code.
- An item over ten working days is escalated to the operations head automatically.
The ageing list did more to improve service than any model change, because it made the forgotten cases visible.
First-time-right and rework
Speed is easy to buy with errors. A team under pressure to clear a backlog can accept fields it should check and close cases it should not. The two quality measures exist to stop that.
First-time-right asks whether a case went through without correction. At Swartland it was measured from the case record described in Every automated recommendation needs a case file: no field changed after indexing, no reallocation, no second letter to the counsellor.
Rework asks what came back. Each reopened case carried the error class from part 4, so the report could separate rework caused by the model from rework caused by intake, rules or review.
Together they make a simple guard. If cycle time falls and first-time-right falls with it, the process has become faster at being wrong.
Cost per completed case
The cost measure follows the method in Cost per prompt is the wrong metric: measure cost per accepted outcome, applied to an operational case rather than a model response.
The numerator is everything the process consumed in the period: OCR and model charges, platform and storage, the staff time of indexers, exception handlers, reviewers and adjudicators, the sampling programme, and the time spent on rework. The denominator is the number of cases completed correctly, not the number of documents processed.
Two effects become visible that a per-document cost hides. First, a model that is cheaper per call but produces more exceptions can raise the cost per completed case, because exceptions are handled by people. Second, rework counts twice: once when the case is first handled and again when it is put right, while it adds nothing to the denominator. Swartland's illustrative figures showed that the people handling exceptions accounted for most of the cost per case, even after automation. That made exception reduction, not cheaper inference, the main cost lever.
Compare the result with the cost of the same case before the AI, measured the same way. A cost per case without a comparison is a number, not a finding.
Prove the change against a baseline
Measures only become evidence when they are compared with something. Swartland captured eight weeks of baseline data before the pilot, using the same definitions it would use afterwards. That discipline is set out in Define the outcome before you build the assistant.
A before-and-after chart can still mislead. Debt-review volumes vary with the calendar, with the economy and with the behaviour of a few large counselling firms. Where it could, Swartland compared like with like: the same weeks of the previous year, the same document types, and for a period, one team using the new process while another still worked manually. When the improvement was reported to the board, it went into the benefits ledger described in From AI portfolio to benefits ledger, with its method and its uncertainty stated.
That brings the series back to where it started. The public Absa case in part 1 reported its figures without stating in full how they were defined or measured, which is normal for a media release. Inside your own organisation you can do better, and you should.
A monthly scorecard
| Measure | How it is defined | Warning sign |
|---|---|---|
| Arrivals and backlog | Items in, items open at week end, per type | Backlog rising while arrivals are flat |
| Age | 90th percentile and oldest item, in working days | Oldest item older than the escalation limit |
| Cycle time | Arrival to decision, and arrival to closure | Indexing faster but cycle time flat (part 3) |
| First-time-right | Cases with no correction, reallocation or re-contact | Falling while cycle time improves |
| Rework | Reopened or corrected cases, by error class | One class rising for two months |
| Exceptions by reason | Share leaving the automated path, by reason code | A new reason appearing, or one source dominating |
| Outcome quality | Sampled error rate with its interval, complaints | Auto-accept band above tolerance (part 4) |
| Cost per completed case | All costs over cases completed correctly | Rising despite falling model cost |
Key points
- A demo measures one clean item. A customer experiences the whole queue. Manage by the queue.
- Track eight measures per document type and per week: arrivals and backlog, ageing, cycle time, first-time-right, rework, exceptions by reason, outcome quality and cost per completed case.
- Lead with the 90th percentile and the oldest item. Averages hide the cases that wait longest.
- Pair speed with quality. If cycle time and first-time-right fall together, the process is getting faster at being wrong.
- Divide cost by cases completed correctly, not documents processed, and compare it with the same measure before the change.
- Capture a baseline with the same definitions before the pilot, and report improvements with their method.
This is part 5 of AI That Changes the Work, Not Just the Interface, and the final part. Previous: Human-in-the-loop is not an evaluation strategy. To start again from the beginning: The most valuable AI in banking may be the AI nobody talks to.
How CloudNala can help
CloudNala helps operations and finance teams measure what AI changes in the back office. We define the queue measures and their warning signs, capture a baseline before the pilot, build the cost-per-completed-case model including the people who handle exceptions, and set up the monthly scorecard and benefits-ledger entry, so that the result you report to your board is one you can defend.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Operational AI Review or write to us at consult@cloudnala.co.za