For most of the past three years, AI has been easy to approve. Large South African organisations, financial services groups in particular, now describe AI, data and digital capability in their results presentations as drivers of growth, efficiency and customer experience. Boards have asked for an AI strategy, and they have received one. Budgets have followed.
That stage is ending. Board members are starting to ask a different question, and it is more uncomfortable than "what is our AI strategy?" It is "which of these things has actually made us better off, and how do we know?"
Many organisations cannot answer it. Not because the AI failed, but because nobody designed a way to find out whether it worked. The cost of each use case is visible on an invoice. The benefit was a forecast in a business case, and nobody has looked at it since the funding was approved.
CloudNala's position is simple. Executive attention is not evidence of value, and neither is usage. Value has to be measured, and measurement has to be designed into the product, the workflow and the operating model before the use case goes live. A benefit claimed after the fact, from data nobody planned to collect, is too easy to inflate and too hard to verify. This series is about the discipline that replaces it: baselines, owners, instrumented journeys, evidence levels, stop rules and a benefits ledger the board can actually manage from.
Indigo Mutual is fictional, and its systems and figures are illustrative. It is a JSE-listed financial services group with life and short-term insurance, an asset manager and a small lending book. Over two years it has committed about R96 million to fourteen AI use cases, from claims document extraction to a staff productivity assistant on every desk.
The year-two question
Indigo Mutual's trouble started in a budget meeting. The Chief Data and AI Officer asked the board's technology and risk committee for a second year of AI funding. The group CFO, who had supported the first year, asked a reasonable question: of the fourteen use cases, which ones had produced the benefits in their business cases?
The answer he received was a dashboard. It showed 3,900 active users, 410,000 prompts a month, 4,500 licences assigned and six models in production. Every number was accurate. None of them answered the question.
When the CFO's team went looking for the benefits, they found three patterns that will be familiar to anyone who has run a technology portfolio:
- No baseline. Several business cases forecast a percentage improvement in handling time or processing cost, but nobody had recorded what handling time or processing cost was before the change.
- No owner. The sponsor who signed the business case had moved roles or left. The delivery team had moved on to the next use case. Nobody owned the benefit.
- Overlapping claims. Three separate business cases each claimed savings from the same contact centre. Added together, they promised more capacity than the contact centre had to release.
None of this meant the AI was useless. Claims document extraction, for example, was plainly saving assessors time. But "plainly" is not evidence a CFO can take to a board, and without evidence, the useful use cases and the expensive ones looked the same.
Cost is visible. Benefit is assumed
There is a structural reason this happens, and it is not carelessness.
Costs arrive with paperwork. A model provider sends an invoice. A cloud bill itemises compute. Licence counts are in a procurement system. Delivery effort shows up in timesheets and supplier invoices. Finance can see cost without anyone having to design anything.
Benefits do not arrive with paperwork. Nobody sends an invoice for forty seconds saved on a call or a fraud loss that did not happen. A benefit exists only if someone defined it, measured the starting point, built a way to observe the change and kept looking. If that work is not done, the benefit stays a forecast forever.
This creates a dangerous asymmetry. AI cost is accurate and growing. AI benefit is optimistic and unverified. Over time the portfolio drifts towards whatever was easiest to approve, not what created value. And because AI is billed as an operating cost that recurs every month, the drift compounds. The Why Do We Keep Paying for AI? series explains why AI is an operating cost, not a once-off purchase. This series picks up the other side of the ledger: proving what that operating cost buys.
Approval is where benefits tracking begins
The most important moment for measuring an AI benefit is not after go-live. It is at approval, when the organisation still has influence over the sponsor and the design is still open.
Indigo Mutual's committee now refuses to fund a material use case unless the business case answers nine questions.
| Control | Required answer |
|---|---|
| Strategic outcome | Which business objective does this use case support? |
| Accountable owner | Which executive owns the realised benefit, by name? |
| Baseline | What is current performance, measured, not estimated? |
| Target | What change is expected, by when, and within what range? |
| Measurement | Which event, system or dataset will provide the evidence? |
| Counterfactual | How will we separate the AI's effect from other changes? |
| Unit economics | What does each successful outcome cost? |
| Risk boundary | Which failures or control breaches force a pause or withdrawal? |
| Review decision | When will we decide to scale, redesign or stop? |
Two of these deserve a comment because they are the ones most often skipped.
The counterfactual is the hardest. Handling time can fall because of AI, or because a new product launch ended, or because the team was restructured, or because it is December. If nothing separates those effects, the AI use case will claim all of them. A counterfactual does not need to be a randomised trial. It can be a comparison group of teams that have not yet received the tool, a staggered rollout by region, or simply a written statement of other changes in the same period so that the claim can be discounted honestly.
Unit economics forces the conversation from "how much does the AI cost?" to "how much does each good outcome cost?" A claims extraction model that costs R4 per document is cheap if it saves twelve minutes of assessor time and expensive if assessors re-check every field anyway. The unit that matters is the accepted outcome, not the model call.
Filling in nine answers takes a sponsor a few hours. Reconstructing them after go-live usually proves impossible, because the baseline has already been overwritten by the change.
Instrument the value chain
Measurement has to exist at more than one level, because value travels through a chain and can be lost at any link.
A use case can succeed at one link and fail at the next. The staff assistant can be heavily adopted and still not change any workflow. Claims extraction can perform well on accuracy and still leave cycle time untouched if assessors are not allowed to trust the extracted fields. Each link therefore needs its own measures.
| Level | Example measures |
|---|---|
| Adoption | Eligible users, active users, repeat use, share of the workflow covered |
| Model and agent | Task success, groundedness, error rate, escalation rate, latency |
| Workflow | Cycle time, exception rate, rework, throughput, completion |
| Customer | Effort, first-contact resolution, conversion, satisfaction, abandonment |
| Financial | Revenue contribution, loss avoided, cost per outcome, payback period |
| Risk and control | Policy breaches, harmful output, unauthorised actions, incidents |
| Resilience | Availability, recovery time, fallback use, dependency failures |
These layers prevent the most common reporting error in AI programmes: presenting high usage as proof of business value. Usage is the first link. The board is paying for the last one. Part 3 shows how to build the business-level measures into a customer journey so the chain can be traced end to end.
Notice also what sits across the whole chain in the diagram: cost, risk, evidence quality and one accountable owner. Governance is not a separate box at the side of the programme. It travels with every link.
Apply the same discipline to vendors
Cloud providers, software platforms and model vendors have a commercial interest in consumption rising. That is not a criticism; it is how their pricing works. But it means the buying organisation has to prove, for itself, whether that consumption improves a business outcome.
A vendor case study is useful. It shows what is possible in someone else's organisation, with their baseline, their process and their data. It is not proof that Indigo Mutual will realise the same value. Part 7 sets out a simple transfer test for vendor claims and a way to tell competitive advantage from increased spend.
Give the board a decision dashboard
Boards do not need prompt counts. They need a portfolio view that shows, for every material use case, the investment, the realised value, how confident the organisation is in that value, the risk exposure, the accountable owner and the next decision.
The most useful single change Indigo Mutual made was to stop reporting one benefit figure. Its board pack now separates:
- forecast benefit, from the business case;
- benefit supported by operational evidence against the baseline;
- benefit independently verified by finance or internal audit;
- cost incurred to date;
- unresolved risk; and
- the scale, redesign or stop decision for the next quarter.
The first time the committee saw the three benefit columns side by side, the gap between them was the conversation. Part 5 covers the dashboard in detail.
The accountability test
Here is the test CloudNala suggests any board applies to its AI portfolio, use case by use case:
- Can we name the baseline?
- Can we name the person who owns the benefit?
- Can we name the system that produces the evidence?
- Can we state the cost of one successful outcome?
- Can we state the date and criteria of the next decision?
If an organisation cannot answer those five questions, the use case is not being managed as a strategic investment. It is being managed as technology consumption. That may be acceptable for small, low-risk tools. It is not acceptable for the use cases that carry most of the budget.
None of this requires new technology. It requires a few hours of discipline at approval, some deliberate instrumentation during delivery and a quarterly habit of looking. The organisations that build those habits now will be the ones that can keep funding the AI that works, stop the AI that does not, and explain the difference to their board.
Next: Part 2, The business case was approved. Who is tracking the benefits?. All parts are on the series page.
Background reading: Gary Stocks, "AI's boardroom honeymoon is over", TechCentral, 22 October 2026.
How CloudNala can help
We help executive teams and boards put evidence behind their AI portfolio. That usually starts with a short assessment of current use cases against the nine approval questions, followed by baseline and benefit-definition workshops for the use cases that carry the most money, and a practical measurement design the delivery teams can build into the product. The result is a portfolio the board can decide from, rather than a dashboard it has to take on trust.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Value Review or write to us at consult@cloudnala.co.za