Model selection conversations tend to go one of two ways. The engineers want the most capable model because it makes their demo work. Finance wants the cheapest model because it makes the forecast work. Both sides bring evidence: a public benchmark, a price list, a blog post from someone who switched. Neither side brings the one piece of evidence that settles it, which is how each model performs on the organisation's own task.
The rule we recommend is easy to state and slightly harder to apply: set the quality bar first, measure every candidate against it on your own cases, then use the smallest model that reliably clears the bar. Cost optimisation is a constraint problem with quality as the constraint, not a race to the bottom of the price list.
Tugela Claims Services is fictional, and its systems and figures are illustrative. Tugela's triage agent, from Part 1 and Part 2, drafts routing recommendations for short-term insurance claims. The team had been running it on a mid-tier model since its first pilot and had never formally tested an alternative.
The bar comes before the comparison
A quality bar has to exist before the models are compared. If it is set afterwards, it tends to be drawn wherever the preferred model happens to land.
Tugela's operations manager and two senior assessors agreed the bar in one meeting:
- Routing accuracy of at least 94% on a labelled set of 400 historical claims, measured against the routing an experienced assessor had applied.
- Zero missed must-escalate claims. Forty of the 400 were cases with fraud indicators or values over R250,000 that must always reach a senior assessor. Missing even one fails the model, whatever its overall accuracy.
- Structured output that validates against the recommendation schema on every case, because the case-management system rejects anything else.
The second criterion matters as much as the first. An overall accuracy figure can hide a model that is excellent on easy cases and dangerous on the rare ones. The bar must include the failures that cost the most, not only the average. The broader method for building such an evaluation set is covered in Evals that gate real work.
Plot cost against quality
Tugela ran the 400 cases through three tiers from the same provider family: a small, a mid-tier and a large model. The same prompt, tools and retrieved context went to each, so that only the model varied.
The results, which are illustrative:
| Tier | Routing accuracy | Must-escalate missed | Cost per case |
|---|---|---|---|
| Small | 91.0% | 3 | $0.06 |
| Mid | 95.8% | 0 | $0.27 |
| Large | 96.5% | 0 | $0.54 |
The small tier failed on both counts. The large tier was the best, by 0.7 percentage points, at twice the price. The mid tier cleared the bar with room to spare.
The tempting conclusion is "use the best model, because quality matters". The correct one is that once the bar is cleared, extra quality has to be worth its price in your workflow, and you can test that. For Tugela, 0.7 points of routing accuracy is roughly three more claims in every 400 routed correctly first time. Each of those saves about 25 minutes of rework, as Part 2 showed. On Tugela's volumes that is worth less than the extra model cost. On a different workflow, where a mistake is far more expensive, the same 0.7 points could easily justify the large tier. The evaluation gives you the numbers to decide either way.
Average results hide segments
The small model's failure looked final. But when Tugela broke the results down by claim type, a different picture appeared. On windscreen claims under R8,000, which are about 35% of Tugela's volume, the small model scored 98% with no structural failures. Its errors were concentrated in complex motor and household claims.
That observation is the basis of routing by segment: sending each category of work to the cheapest model that passes on that category.
The router itself is deliberately boring. It is a deterministic rule on claim type and value, not a model deciding how hard a claim is. Model-based difficulty routing has its place, but it adds another component that can be wrong and another thing to evaluate. Where a simple field already separates the segments, use the field.
With windscreen claims on the small tier and everything else on the mid tier, Tugela's blended model cost per case fell by roughly a quarter, with no change in measured accuracy on either segment.
The conditions that make routing safe
Routing by segment saves money only if it is governed as carefully as the original choice. Four conditions matter.
Each lane has its own evaluation slice. The windscreen lane is tested on windscreen cases. A model change in that lane is approved on that slice, not on the overall set.
The router is tested too. A claim misclassified as windscreen when it is a full motor claim lands in the cheap lane. Tugela added router cases to the evaluation set, including ambiguous claim forms.
Escalation is one-way and cheap. If the small model's output fails schema validation or its confidence rule, the case moves to the mid tier automatically. That fallback costs a second call, which should be counted, but it limits the damage of a bad lane.
The bar is re-run on every model update. Providers revise and retire models. A version that passed last quarter may behave differently after an update. Continuous evals are tests for AI systems explains why this has to be routine rather than a project.
When not to optimise
There are cases where the smallest passing model is not worth the engineering effort.
If the whole workload costs a few thousand rand a month, the saving from routing will not pay for the evaluation and router maintenance. Use one model that passes and spend the effort elsewhere.
If volumes are low and stakes are high, such as a handful of legal or medical summaries a day, the cost difference between tiers is trivial next to the cost of an error. Use the strongest model that passes and stop.
If you do not yet have an evaluation set, model choice is guesswork whichever way you go. Build the evaluation first. That is the real prerequisite, and it is the same asset you need for governance, procurement questions and every later model change.
A short decision procedure
- Agree the quality bar with the people who own the outcome, including the must-not-fail cases.
- Build a labelled evaluation set from real historical work, at least a few hundred cases for a workflow like triage.
- Run every candidate model on identical inputs and record accuracy, must-not-fail misses, schema failures and cost per case.
- Choose the cheapest model that clears the bar overall.
- Break results down by segment. Where a cheaper model passes on a segment that can be identified deterministically, consider a separate lane with its own slice.
- Re-run the whole procedure on any model, prompt or router change.
The provider's price list is one input to this procedure. It is never the procedure.
Next: Part 4, Prompt caching is an architecture decision. All parts are on the series page.
How CloudNala can help
We help teams set a quality bar with the business owner, build an evaluation set from their own historical cases, and compare model tiers on cost and quality together. Where it pays, we design segment routing with its own tests and fallbacks, so the saving is real and stays real after the next model update.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Cost Review or write to us at consult@cloudnala.co.za