Fallback routing is usually the feature that sells a model gateway. If the primary provider has an outage, the gateway sends the request to another model, and users never notice. Ukhozi turned it on for every route after the retry incident described in Part 4.
Three weeks later the primary provider had a forty-minute degradation. The fallback worked exactly as designed. Every request got an answer. And the billing-dispute workflow quietly accumulated a backlog of cases it could not process, because the fallback model returned its structured output in a slightly different shape, about one time in twenty (illustrative). The cases did not fail loudly. They sat in an exception queue until someone noticed on Thursday.
That is the trade-off in one incident. CloudNala's position: failover is a per-route decision. A route may fail over only to a model that has passed that route's evaluations, under equivalent data terms, with an identical output contract. Otherwise it should fail closed, clearly.
Ukhozi Telecom is fictional, and its systems and figures are illustrative.
What failover protects, and what it can change
The left-hand column is real. For a staff assistant that answers plan questions, an answer from a different good model is far better than an error during a busy Saturday in the stores.
The right-hand column is also real, and it is easy to miss because none of it shows up as an error:
Different refusal behaviour. Models draw lines in different places. A request the primary model handles may be refused by the fallback, or the reverse. For a customer-facing agent, the reverse is the more dangerous case.
Structured output breaks. Workflows that parse JSON, call tools with specific argument shapes or rely on a model following a strict format are sensitive to model changes. Even with a schema, models differ in how reliably they comply.
Smaller context limit. If the fallback model's context window is smaller than the primary's, long requests fail or are truncated. A truncated prompt can produce a confident answer based on half the evidence.
Data goes to a new place. A fallback on a different provider, or a different region of the same provider, may operate under different data-processing terms and retention. For routes carrying personal information, that is a question for the privacy owner, not a routing default. AI and POPIA, in Plain Language covers the underlying questions.
Cost per request changes. A fallback to a larger model during a long outage can be expensive; a fallback to a smaller one can lower quality.
Three questions per route
Ukhozi replaced its blanket setting with a gate that each route had to pass before failover was allowed.
1. Has the fallback passed this route's evaluations? Not a general benchmark. The same evaluation set the route uses for its primary model, with the same pass threshold. If the dispute workflow's evaluation requires 98% valid structured output, the fallback must meet it too.
2. Are the data-processing terms and region class equivalent? Same contractual commitments on retention and training use, same class of processing location for the data the route carries. If not, the privacy owner decides, in advance and in writing.
3. Is the output contract identical? Schema, tool definitions, maximum lengths and context window. If the fallback cannot accept the same request and return the same shape, the route needs a translation layer and its own tests, or no failover.
A route that passes all three may fail over, and every failover is logged with the route, the reason and the duration, so that quality can be checked afterwards.
A route that fails any question fails closed: it returns a clear error, or degrades to a non-AI path. For Ukhozi's dispute agent the non-AI path already existed: route the customer to the standard dispute form and tell them a person will respond within the usual time. That is a worse experience than the agent and a much better one than a silently stuck case.
How Ukhozi's routes landed
| Route | Failover? | Reason |
|---|---|---|
| Store-support assistant | Yes | Fallback passed its evaluation; no personal information |
| HR policy assistant | Yes, same provider other region | Data terms equivalent, privacy owner agreed |
| Call-centre summaries | Delay instead | Summaries are not urgent; queue and process after recovery |
| Billing-dispute agent | No, fail to the form | Structured output not reliable enough on the fallback |
| Sales-quote drafting | No, fail closed | Pricing logic tested only on the primary |
"Delay instead" is a useful third option that blanket failover hides. Work that is not interactive can wait in a queue for the primary to return, at no quality risk.
Keep the fallback tested
A fallback that is never used drifts. Its model version changes, its rate limits are lower than you thought, or its credentials expired. Ukhozi added three habits:
- The evaluation set for every failover-enabled route runs against the fallback weekly, not only when it was first approved.
- A small share of real traffic (a few percent) is routed to the fallback continuously on low-risk routes, so that it is exercised and monitored.
- Each quarterly game day includes a forced failover, checked for quality as well as availability.
The short version
Failover trades consistency for availability. Decide that trade per route.
Three questions. Evaluated, equivalent data terms, identical contract.
Failing closed is a design choice. A clear error or a non-AI path can be the safer answer.
Delay is an option. Non-interactive work can wait for the primary.
Exercise the fallback. An untested fallback is a guess.
Previous: Part 4, who owns the retry? Next: Part 6, do your observability tools store the prompt? All parts are on the series page.
How CloudNala can help
We review fallback configurations route by route: which routes can safely fail over, which need a translation layer or a non-AI path, which data terms must be checked, and how fallbacks are kept tested. The outcome is a routing policy that keeps services available without quietly changing what they do.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za