Ask an AI platform team how their system is doing and you will get a precise answer. Availability was 99.9% last month. Median latency was 1.8 seconds. The claims model processed 41,000 documents and consumed so many tokens at such a cost. Error rates are down.
Ask the same team whether the system has made claims faster for customers and the answer becomes vague. Probably. Assessors say it helps. Nobody has looked at the end-to-end claim time.
This gap is not a failing of the team. It reflects what comes free and what has to be built. Every modern AI platform, model gateway and cloud service emits technical telemetry automatically. Nothing emits business telemetry automatically. If a board wants to know whether an AI use case changed a customer outcome, someone has to design the measurement into the journey.
CloudNala's position is that instrumentation is part of the use case, not part of the platform. The technical measures tell you whether the system is healthy. Only the business measures tell you whether it is worth having. Both are needed, and they are only useful together if they can be joined.
Indigo Mutual is fictional, and its systems and figures are illustrative. In Part 2 its CFO discovered that several AI benefits had no baseline and no owner. This part looks at one of its better use cases, claims document extraction, and what it took to prove that it worked.
Two layers of measurement
The bottom layer is what platforms report: availability, latency, error rate, tokens consumed and cost per call. These measures matter. A use case that is slow or unreliable will not be adopted, and one that costs more per call than expected will erode its own business case. Engineering teams should watch them closely, as covered in Observability and cost: why production AI must be visible.
The top layer is what the board funds: completion rate, conversion, handling time, loss avoided, customer effort and revenue. These measures live in business systems, not AI systems: the claims platform, the CRM, the contact-centre software, the general ledger.
The two layers fail in characteristic ways when they are not joined. A technically excellent model can produce no business change because the workflow around it did not change. A business improvement can be attributed to the AI when it was caused by something else. Only a join, case by case, can tell the two apart.
The join is a shared identifier
The practical mechanism is unglamorous: every event in the journey, technical or business, carries the same case or journey identifier.
When Indigo Mutual looked at its claims extraction use case, the model calls were logged with a request ID generated by the AI service. The claims platform recorded claims with a claim number. Nothing linked the two. The AI team could say how many documents had been processed; the claims team could say how long claims took; nobody could say whether claims with AI-extracted documents moved faster than claims without.
The fix was to pass the claim number into every AI call as metadata and to write it into the model gateway's logs. From that point, any claim could be traced from lodgement through extraction, review and decision to payment, with both the technical and business events on one timeline.
This sounds trivial. In practice it touches several teams: the channel team that creates the case, the AI team that calls the model, the platform team that runs the gateway and the data team that builds the reporting. It needs to be in the use case's design from the start, which is why Indigo Mutual now treats "which identifier joins the layers?" as one of the measurement questions at approval.
Instrument the journey step by step
The second change was to define the business events the journey emits, not just the outcome at the end.
Each step emits an event with the claim reference and a timestamp. Six events, each answering a different question:
- claim_lodged gives volume and the start of the clock.
- docs_added shows customer effort: how many uploads, how many attempts, how long between them.
- data_extracted records what the AI produced and, importantly, its confidence and which fields it filled.
- review_closed records what the assessor changed. This is the most valuable event in the chain, because the difference between what the AI extracted and what the assessor accepted is a direct measure of accuracy in production, and the time the assessor spent is a direct measure of rework.
- decision_made gives cycle time and the decision itself.
- payment_sent closes the outcome.
With these six events, Indigo Mutual could answer questions the dashboard never could. Claims where extraction confidence was high and the assessor changed nothing closed 1.6 days faster on average than comparable claims before the tool. Claims where the assessor corrected more than three fields took longer than before, because checking and fixing was slower than typing from scratch. That second finding was the useful one: it showed exactly which document types the model should not handle yet.
Measure what the human did with the output
The review step deserves emphasis because it is where most AI value is either realised or lost, and it is the step most often left uninstrumented.
If assessors accept extracted fields without checking, the time saving is real but the accuracy is untested. If they check every field, accuracy is protected but the time saving may vanish. If they correct a lot, the model may be making work rather than saving it. All three situations look identical on a technical dashboard: the model ran, it returned output, the call succeeded.
Recording what the human accepted, changed and rejected, and how long that took, turns the review step into a continuous evaluation. It also gives the benefit owner from Part 2 the evidence they need. "Handling time fell" becomes "handling time fell on these document types, rose on these, and here is the case-level data."
The same idea applies outside claims. A contact-centre agent-assist tool should record whether the agent used the suggested answer, edited it or ignored it. A fraud-alert triage model should record whether the analyst agreed. A sales assistant should record whether the drafted proposal was sent largely as written. Each of those is an event the business system can capture cheaply if someone designs it.
What to instrument first
Instrumenting everything is expensive and rarely necessary. Indigo Mutual uses a short rule for each new use case:
| Question | Instrument |
|---|---|
| Where does the AI step start and end? | Events either side of it, with the case ID |
| Where does a human accept or change the AI's output? | The accept, edit and reject actions, and time spent |
| What outcome does the business case promise? | The event that proves it, from the business system of record |
| What could cause the same outcome without AI? | A comparison group, a staggered rollout or a dated log of other changes |
| What does one outcome cost? | Model and platform cost joined to the outcome event |
The last row connects this part to the economics. Once AI cost is joined to the business outcome by case ID, cost per accepted outcome becomes a simple query rather than an estimate. The Economics of AI Delivery series goes deeper into that measure.
Boundaries
Journey instrumentation creates data about customers and staff. Event streams that record what an assessor changed or how long an agent took are records about people, and POPIA applies to them. Collect what the measurement needs, set a retention period and decide in advance who may see individual-level data. Measuring the system is legitimate; turning the instrumentation into a surveillance tool for individual staff is a different decision that needs its own justification.
It is also worth being honest about precision. A comparison between claims before and after the tool will always have noise in it. The goal is evidence good enough to support a decision, not a laboratory result.
Next: Part 4, AI consumption is not AI value. All parts are on the series page.
How CloudNala can help
We help delivery and data teams design journey instrumentation for AI use cases: the events, the joining identifier, the review-step capture and the comparison approach, sized to what the business case needs to prove. Where a model gateway or observability platform is already in place, we connect its technical telemetry to the business events so cost per accepted outcome can be reported directly.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Value Review or write to us at consult@cloudnala.co.za