It is the last Friday of the month, a little after five. Payday traffic is at its peak, and a shopper in Pietermaritzburg types into the meal assistant: "supper for four tonight, my son is allergic to peanuts, under R400, we have rice". At that moment the model deployment that normally writes the meal plans starts returning rate-limit errors. In a few seconds she will either have a basket she can review or an error she will remember the next time she shops.
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery. Its people, systems and figures are illustrative and do not describe any single organisation.
Part 9 showed that tokens are a capacity dependency with hard limits. This part is about what happens when the journey hits those limits, or when a model endpoint fails outright, while a shopper is waiting. The outcome at risk is plain: a shopper who gets an error on payday evening has been lost to the agent, and sometimes to the store.
Route by task, not by shopper
Ubuntu Grocer's meal assistant is not one model call. The journey orchestrator, a deterministic workflow, calls a sequence of bounded agent steps with different jobs. Deciding whether the shopper wants a meal, a product or help with an order is a short classification. Rewriting "amasi and morogo for the weekend" into catalogue search terms is a reformulation. Re-ranking twenty candidate products for one ingredient line is a comparison against explicit attributes. Turning "supper for four, peanut allergy, under R400, we have rice" into a sensible plan is open-ended reasoning.
Only the last one needs the strong model.
Our article on routing agent tasks away from the most expensive model treats routing as a sequential decision over a session, with hard gates before any router and cost measured per accepted outcome. All of that applies here; the retail journey adds a simplification and a complication.
The simplification is that the boundary is already drawn. A workflow with bounded agent steps, the design chosen in Part 6, gives each step a typed input, a typed output and an owner, so the step is the unit you route, and it is tested on its tier, as our article on using the smallest model that passes the evaluation recommends.
The complication is that the journey is interactive, commercial and full of hard constraints. When the strong tier fails at five on a Friday, waiting is not an option, and sending the step somewhere cheaper can be dangerous.
Shoppers first, batch work last
Not all model traffic comes from shoppers. Overnight jobs write recipe descriptions and suggest catalogue synonyms on the same deployments and quota.
The first rule is that interactive journey traffic has priority over batch work, enforced at the AI gateway that sits in front of every model endpoint: separate quota allocations for interactive and batch consumers, a batch queue that backs off when interactive latency rises, and batch requests shed first when an endpoint signals overload. A recipe description written at 02:00 instead of 17:00 costs nothing. A meal plan that fails at 17:00 costs a basket. Write the priorities down and have the product owner sign them, or the gateway's default, usually first come first served, decides for you.
Circuit breakers and capacity in more than one place
Retrying a failing endpoint harder makes it worse. A circuit breaker opens after failures cross a threshold within a window, so that calls fail fast instead of piling on; after a timeout it lets a few trial calls through (the half-open state) and closes only if they succeed. Martin Fowler's CircuitBreaker is the classic description, and the Azure Architecture Center's version covers rate limits and multiple regions.
For model endpoints, two details matter.
Rate limits are not outages. An endpoint over its quota typically returns HTTP 429 with a Retry-After header, and that value can be long. The breaker should honour it. Who owns the retry? explains why exactly one layer, usually the gateway, should own retries and breakers.
One breaker per deployment, not per model. Ubuntu Grocer gives its strong tier capacity in more than one location or provider deployment, precisely so that one breaker can open while another path stays closed. Gateways commonly express this as priority groups. Azure API Management's backend pools, for example, document priority-based load balancing in which lower-priority backends are used only when every higher-priority backend is unavailable because its circuit breaker has tripped (checked on 5 November 2026).
A second location is not free: it may add latency or sit under different data-processing terms, and it must have passed the same evaluation as the first. Failover can improve availability and damage consistency sets out the per-route gate: same evaluation, equivalent data terms, identical output contract.
The time budget decides what failover is possible. Say, for illustration, the planning step has eight seconds. If the gateway waits seven for the primary to time out, the fallback you designed never runs in practice. So the budget is split in advance: in our worked example the primary gets two and a half seconds to its first token and five in total, an open breaker sends the call straight to the second location without waiting, and anything that cannot finish in the time left drops a rung on the ladder below.
A fallback that answers is not a fallback that works
Ubuntu Grocer's team rehearsed this failure in a game day. The strong tier was forced unavailable, and the gateway sent planning to a smaller model with a shorter prompt template. The smaller model proposed chicken satay for four. The plan was well formed, so it passed the schema check. Then the hard-constraint filter, which runs in code on the final product set as Part 4 requires, removed the peanut sauce, the peanut butter and a spice mix that declares peanuts. The basket on the shopper's screen was chicken, skewers and no sauce. Nothing unsafe was proposed, but the plan was incoherent. In a worse variant, with an allergen attribute missing rather than set, it would have been unsafe.
Two rules came out of it.
Validate the plan against the constraints, not only the basket. Ruan Botha, the AI engineering lead, added a check between planning and matching: if the filter would remove an ingredient central to the recipe, the plan is rejected, not patched.
Unknown is not safe. A product whose allergen attribute is missing is excluded for a shopper with that allergy, on every path. Sipho Nkosi's catalogue team owns those attributes, so a missing value is a catalogue defect with a named owner, not a model problem.
The checks are cheap and deterministic: the schema is valid, every ingredient maps to at least one candidate, no central ingredient is filtered out, and the estimated basket is within budget. A fallback output that fails any of them is discarded.
Degrade one rung at a time
When the strong tier is down and the fallback fails its checks, the journey still has useful things to offer. Ubuntu Grocer wrote them down as a ladder, so nobody improvises during an incident.
The rungs remove capability in order of least harm. The second location gives the same result a little slower. A smaller model working from curated, pre-approved recipe templates gives a simpler plan. With no model at all, rules can still select approved recipes that pass the shopper's filters and show a plain, non-personalised ingredient list she can add herself. At the bottom, the normal store search and basket work as they did before the agent existed, and that rung must not depend on any agent service.
The column on the right is the point of the diagram. The hard-constraint filter is not a rung. It runs on every path, including the rules-only path, because a fallback that quietly drops the peanut allergy is not degraded service. It is a different and dangerous product.
The ladder carries one asymmetry that teams often miss: degrade on model dependencies, fail closed on constraint dependencies. If the model is down, show less. If the service that supplies allergen and dietary attributes is down, stop proposing products for that shopper and say so plainly. Pieter van Wyk, the principal architect, wrote it into the decision record in one line: no path may propose a product whose constraints have not been checked.
A degraded answer should also look degraded. "Our meal planner is busy, so here are three peanut-free suppers for four under R400" is honest and useful. Every rung change is also an operational event, recorded with the endpoint that served the call and the reason. Part 11 attaches those events to one basket's trace, so Ayesha Patel's site reliability engineering (SRE) team can see how many payday shoppers got the full plan.
Questions to take into your architecture review
- Which agent steps run on which model tier, and where is the evaluation evidence that each step passes on its tier?
- How does the gateway give shopper traffic priority over batch work, and what is shed first under pressure?
- Where do circuit breakers sit, what trips them, and do they honour
Retry-Afteron rate-limit responses? - In how many locations or deployments does each critical tier have capacity, and has each passed the same evaluation and data-terms check?
- Does every fallback fit inside the time that remains in the step's budget when it triggers?
- Which deterministic checks must a fallback output pass before the shopper sees it?
- What does the shopper see on each rung, and does the bottom rung work with every agent service switched off?
- What happens to a shopper with an allergy if the constraint data service is unavailable?
Key points
- Route each bounded agent step by task class. Only open-ended planning earns the strong model.
- Give shopper traffic priority over batch work, and let one layer own retries and circuit breakers.
- Capacity in more than one location helps only if every path passed the same evaluation and fits the time budget.
- Check fallback quality with cheap deterministic rules, and reject a plan the constraint filter would hollow out.
- Degrade on model dependencies; fail closed on constraint dependencies. The allergen filter runs on every rung.
CloudNala's point of view
Most failover designs we review are written and tested for availability, so they answer "did the shopper get a response?" and never "did the shopper get a safe, coherent basket?". In retail, the second question is the one that matters. We would rather see a team ship three well-tested rungs than a sophisticated multi-location router whose fallback has never met a real allergy. A degraded journey that tells the truth keeps the customer; a confident one that drops a constraint can lose them for good.
Next: Part 11, Trace one basket across every dependency. All parts are on the series page.
How CloudNala can help
CloudNala reviews routing and failover for customer-facing agents step by step. We map each agent step to a tier with evaluation evidence, set interactive and batch priorities at the gateway, design breakers and time budgets so that failover can happen inside the shopper's wait, write the deterministic checks a fallback must pass, and agree a degradation ladder with your product and safety owners. Then we run a game day that forces each rung and checks the basket, not just the status code.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za