At 10:40 on a Tuesday, Ukhozi's primary model provider started returning intermittent errors. The provider's status page called it "elevated error rates". Within five minutes Ukhozi's own traffic to the provider had risen nearly tenfold, its rate limit was exhausted, and requests that would have succeeded were being refused. When the provider recovered at 11:15, Ukhozi's billing-dispute agent kept failing for another twenty minutes while queued retries drained.
Nothing was misconfigured in the usual sense. The application made up to three attempts at a failed request. So did the agent framework. So did the gateway. So did the provider's SDK inside the gateway. Every team had chosen a reasonable setting. Nobody had chosen the combination.
CloudNala's position: for every call path, exactly one layer owns retries, every inner timeout is shorter than the one around it, and no call with a side effect is retried without an idempotency key.
Ukhozi Telecom is fictional, and its systems and figures are illustrative.
How retries multiply
Retries nest. If the outer layer retries a call, every inner layer starts its own retry cycle again.
With four layers each allowed three attempts, one user request can produce up to 81 provider calls in the worst case. Real numbers are usually lower, because some errors are not retried and timeouts cut cycles short. But the direction is the problem: the more the provider struggles, the more traffic it receives from you. That is a retry storm, and it turns a partial outage into a full one for your organisation, and sometimes for the provider's other customers too.
Three further effects follow.
Cost. Any attempt that reaches the provider and completes is billed, including attempts your side has already abandoned after a timeout. Long prompts make that expensive: an agent with a 30,000-token context whose retries complete nine times has paid for 270,000 input tokens to produce one answer.
Latency. Each retry adds its own wait. A user who would have seen a clear error in five seconds instead waits ninety seconds and then sees an error anyway.
Duplicate actions. A model call is usually safe to repeat. A tool call often is not. If the agent framework retries a whole step that includes "open a dispute and send the customer an SMS", the customer may get two SMS messages and the billing system two disputes.
Classify the error before retrying
Not every failure deserves a retry. A useful rule of thumb, which Architecture in Plain English covers in more depth for HTTP status codes generally:
| Failure | Retry? | Why |
|---|---|---|
| Rate limited (HTTP 429) | Yes, after the delay the response asks for | The provider has told you when to come back |
| Provider overloaded or unavailable (502, 503, 529) | Yes, limited, with backoff and jitter | Often transient, but retries add load |
| Timeout on your side | Once at most, if the call is safe to repeat | The provider may still have processed it |
| Bad request, auth failure (400, 401, 403) | No | Repeating a wrong request produces the same error |
| Content policy refusal | No | It is an answer, not a fault |
| Context too long | No | Fix the request, or route to a model with a bigger window |
Backoff means waiting longer between attempts; jitter means randomising that wait so a thousand clients do not retry at the same instant.
Choose one owner
For model calls behind a gateway, the gateway is usually the right owner. It sees all traffic to the provider, so it can apply a shared budget for retries, honour the provider's rate-limit headers centrally, and stop retrying everything when the provider is clearly down. That last capability is a circuit breaker: after a threshold of failures, the gateway fails fast for a short period instead of adding load, then tests the provider again.
Ukhozi's revised design:
- Application: no automatic retry of model calls. It shows a clear message and lets the user try again.
- Agent framework: no retry of model calls. It may retry a whole step only if every tool call in that step is idempotent.
- Gateway: the retry owner. Two attempts with backoff, then one fallback attempt if the route allows it (Part 5), within a total budget.
- Provider SDK: retries switched off inside the gateway, so the gateway's settings are the real ones.
Make timeouts nest
The second half of the rule is timeouts. Each layer's timeout must be shorter than the timeout of the layer that called it, with room for its own retries. Otherwise the outer layer gives up while the inner one is still working, the user sees an error, and the inner layer completes the call anyway and bills for it.
Ukhozi's illustrative budget for the dispute agent: the user request has 20 seconds; the application waits 18; the framework 16; the gateway allows two attempts of 6 seconds plus a fallback within its share. The numbers matter less than the property: inner always shorter than outer, and every layer knows its budget.
Streaming changes the arithmetic. A streaming response can take 40 seconds legitimately while producing tokens steadily. Ukhozi used a time-to-first-token timeout and an idle timeout between chunks for streaming routes, rather than a single total timeout.
Protect side effects
Retries in the model layer are a cost and load problem. Retries in the action layer are a correctness problem.
Every tool call that changes something should carry an idempotency key, a unique value derived from the business event (for example, the dispute request and the bill line), so that the receiving system can recognise a repeat and return the original result instead of acting twice. Ukhozi's billing API already supported this for its web channel. The agent simply had not been sending it. The Agent Evidence Layer shows the same fix for a credit-note tool.
What changed after the Tuesday
With one retry owner, nested timeouts, a circuit breaker and idempotency keys on the two write tools, Ukhozi ran a game day that recreated the provider failure. Peak traffic to the provider during the simulated outage rose by about 40% rather than tenfold (illustrative), users got a clear error within the budget, and the agent recovered within a minute of the provider returning.
The short version
Retries multiply across layers. Four reasonable settings can make dozens of calls.
Classify before retrying. Some errors are answers, not faults.
One layer owns the retry. Usually the gateway, with a circuit breaker.
Timeouts nest. Inner always shorter than outer.
Side effects need idempotency keys. Otherwise a retry repeats the action.
Previous: Part 3, virtual keys are an accountability mechanism. Next: Part 5, failover can improve availability and damage consistency. All parts are on the series page.
How CloudNala can help
We trace retry and timeout settings across every layer of an AI call path, identify where they multiply, and design a coordinated policy: a single retry owner, nested timeout budgets, a circuit breaker, and idempotency on the actions that matter. Then we help you run a game day that proves the design holds when a provider degrades.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Platform Architecture Review or write to us at consult@cloudnala.co.za