Every AI vendor has a case study. A bank cut onboarding time by some impressive percentage. An insurer automated a large share of its claims. A retailer's customer service costs fell sharply. The figures are usually real, in the sense that something like them happened at the named customer under particular conditions.
The question for a buyer is not whether the case study is true. It is what the case study is evidence of. And the honest answer is usually: evidence that the outcome is possible, in that organisation, with that baseline, that process and that data. Not evidence that it will happen in yours.
There is a second, harder question. Even when the outcome does transfer, is it an advantage? If every competitor can buy the same tool from the same vendor and get the same result, the organisation has not gained ground. It has kept pace, and paid more to do it. Keeping pace can be necessary. It should just not be reported to a board as strategy.
CloudNala's position is that vendor evidence is a reason to run a measured pilot, never a substitute for the organisation's own baseline and measurement, and that consumption bought from a vendor is only an advantage when it is combined with something competitors cannot buy.
Indigo Mutual is fictional, and its systems and figures are illustrative. Earlier parts built its benefit records, journey instrumentation, board dashboard and stop criteria. This part looks at how its committee now treats vendor proposals and claims of strategic advantage.
Why vendor incentives matter
This is not a criticism of vendors. Model providers, cloud platforms and software companies price on consumption, seats or both. Their revenue rises when usage rises. That is a legitimate business model, and many of them genuinely want customers to succeed, because successful customers buy more.
But it does mean the vendor's measure of success is consumption, and the buyer's measure of success has to be outcome. Those overlap only partly. A vendor can be delighted by a deployment that doubles token consumption without changing any business result. A buyer should not be.
It also explains why vendor dashboards, as Part 4 discussed, show usage so prominently, and why vendor business cases tend to quote the most favourable reference customers. None of this is deceptive. It is simply not the buyer's evidence.
The transfer test
When a vendor's case study is used to justify an AI investment at Indigo Mutual, the committee asks four questions of it.
Same baseline as ours? A 40% reduction in handling time from a starting point of fourteen minutes is a different achievement from 40% off seven minutes. If the reference customer's starting point was much worse than Indigo Mutual's, the percentage will not transfer.
Same process and exceptions? Case studies are often drawn from the cleanest part of a process. If the reference customer automated simple motor claims and Indigo Mutual's volume is dominated by complex commercial claims, the result does not apply.
Same data quality? AI outcomes depend heavily on the documents, records and history the model works with. A reference customer with digitised, structured records will see different results from one working with scanned forms and free text. Data foundations before AI covers why this dominates outcomes.
Measured by whom, and how? Was the outcome measured by the customer's finance function against a baseline, or estimated by the vendor's customer success team? Over what period? Including or excluding the human review effort?
If the answers are unknown or unfavourable, the case study goes into the "possibility" box: a reason to run a measured pilot with a proper baseline, as set out in Part 1. Only the organisation's own measured results go into the "proof" box. That is the box the business case and the board dashboard draw on.
Consumption or advantage?
The second test is strategic rather than evidential. Suppose a vendor's tool genuinely delivers its promised result at Indigo Mutual. Has Indigo Mutual gained an advantage?
Who can have it? If the capability is a licence anyone can buy, competitors will have it within a quarter. The saving is real, but the market position has not changed.
Where does value land? For commodity capability, much of the value flows to the vendor as usage revenue, and to customers through prices that competition pushes down. For differentiated capability, value shows up in the organisation's own margins, service levels or risk position.
How long does it last? Commodity advantage lasts until competitors adopt the same tool. Differentiated advantage lasts as long as the thing that makes it hard to copy: proprietary data, a deeply integrated process, specialist knowledge, distribution, or trust.
How is it measured? Commodity capability is usually measured in seats, calls and tokens. Differentiated capability is measured in outcomes against a baseline, because there is no vendor dashboard for it.
Neither column is bad. A staff productivity assistant is almost certainly in the left column, and that may be fine: not having it might be a disadvantage. Claims data extraction combined with twenty years of Indigo Mutual's own claims history, its assessors' corrections and its fraud patterns might belong in the right column, because a competitor buying the same model would not have the same training signal or the same integrated workflow.
The mistake is reporting the left column as if it were the right one: calling a widely available licence a strategic investment.
Where differentiation usually comes from
When Indigo Mutual looked across its fourteen use cases for genuine advantage, the candidates shared three features:
- Proprietary data the model works with. Not data the vendor has, but the organisation's own records, history and outcomes, governed well enough to use.
- Deep workflow integration. The AI changes how a core process runs, end to end, in a way that took real design work and would take a competitor time to replicate.
- Feedback loops the organisation owns. Human corrections, outcomes and exceptions flow back into improving the system. Over time, the system gets better at this organisation's specific work in a way a generic tool does not.
The models themselves were rarely the source. Model capabilities move fast and are available to everyone. What the organisation builds around them is not.
Negotiating with the evidence
The transfer test also changes vendor conversations. When Indigo Mutual asks a vendor for the reference customer's baseline, measurement method and review effort, good vendors provide it, and the conversation becomes more useful for both sides. When a vendor cannot or will not, that is information too.
It also changes contracts. Where a vendor is confident in an outcome, Indigo Mutual asks for commercial terms that reflect it: a measured pilot before commitment, exit points tied to evidence, or pricing linked in part to outcomes rather than pure consumption. Not every vendor will agree, but the request clarifies who carries the risk.
So which AI should we actually pay for? offers a complementary view from the budget-holder's side.
Next: Part 8, From AI portfolio to benefits ledger. All parts are on the series page.
How CloudNala can help
We help organisations assess AI vendor proposals independently: applying a transfer test to case-study claims, designing measured pilots with proper baselines and exit points, and separating commodity capability from investments that could build lasting advantage.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Value Review or write to us at consult@cloudnala.co.za