It is the second Sunday of the school holidays, just after payday. A grandmother is planning lunch for nine: bobotie, samp and beans, a vegetarian option for her grandson, and nothing that costs more than it needs to. She types the request into the meal assistant at three in the afternoon, a few minutes after a push notification about a festive special has gone out to every shopper with the app. So have thousands of other people.
Earlier that afternoon her plan would have appeared in a few seconds. Now, with only somewhat more shoppers online, it takes many times longer. Nothing broke. The system crossed a line nobody could see on the dashboard.
Ubuntu Grocer is a composite, a fictional South African grocery retailer. Its scenario draws on lessons from real enterprise retail-agent delivery, but its people, systems and figures are illustrative and do not describe any single organisation. Part 7 built a scenario generator for realistic journeys. This part is about what it teaches when you turn the arrival rate up: why latency bends, and why the bottleneck keeps moving. Every number here is illustrative.
The outcome at risk is the peak itself. Payday Sundays, Black Friday and the week before Christmas are when basket volumes peak, and exactly when a system running near its knee stops converting.
Latency is not linear in load
Every constrained resource behaves roughly like a queue. The useful quantity is utilisation: arrival rate multiplied by service time, divided by the number of workers that can serve in parallel. A model deployment, a pool of orchestrator workers and a database each have one.
In the simplest queueing model, with random arrivals and one server, the average time in the system is service time divided by one minus utilisation. That fraction explains most of what surprises teams under load.
Take an illustrative planning step that needs 2 seconds of service. At 50% utilisation the shopper waits about 4 seconds in total. At 80%, about 10 seconds. At 90%, about 20. Going from 80% to 90% busy costs more extra waiting than going from zero to 80%. That bend is the knee, and beyond it small increases in traffic produce large increases in delay.
Three refinements make the picture more realistic, and none of them is comforting.
Variability makes it worse. The more service times vary, the longer the queues at the same utilisation. Large language model (LLM) calls vary a great deal, because generation time grows with output tokens: a short classification and a week's meal plan share the same queue.
More workers move the knee, and sharpen it. A large pool can run at higher utilisation before queues form, which is why scaling out helps. When it does saturate, delay rises more abruptly.
Tails grow faster in absolute terms. The 95th-percentile (p95) latency is the time within which 95% of requests complete. In this simple model it is about three times the average, so as the average stretches, the p95 crosses a shopper's patience first.
Journeys multiply the tail
A journey is a chain of steps, some sequential and some fanned out in parallel. Sequential steps add. Parallel fan-out waits for the slowest branch: the ingredient-to-product matcher looks up every ingredient at once, and the turn waits for the last lookup. Suppose, for illustration, that each lookup has a 1% chance of being unusually slow. For a twelve-ingredient Sunday lunch, the chance that at least one is slow is one minus 0.99 to the power of twelve, about 11%. For a forty-ingredient week of meals, about 33%. As Part 1 noted, the journey is only as fast as its slowest dependency; the fan-out arithmetic shows how fast that bites. A tail that is rare per call becomes common per journey. Dean and Barroso's The Tail at Scale makes the same argument for large interactive services and sets out tail-tolerant techniques.
So per-service dashboards, each showing a healthy p95, can coexist with a poor journey p95. Measure the journey too, which is why Part 7 judged runs on time to basket.
The bottleneck moves
When Ubuntu Grocer ran its stepped ramp, the first constraint was not the last. In reviews of retail-agent programmes we see a typical order in which limits surface, though the exact order depends on how each system is built and provisioned.
1. Gateway quota and model rate limits. Providers express limits per minute, and some enforce them over shorter windows. Microsoft's Azure OpenAI documentation says request rates are evaluated over "a small period of time, typically 1 or 10 seconds" (Microsoft Learn), and Anthropic's notes that a limit of 60 requests per minute "might be enforced as 1 request per second" (Claude documentation); both checked on 5 November 2026. A push-notification burst can therefore produce HTTP 429 (too many requests) responses while the per-minute average still looks comfortable. Part 9 turns this into a capacity model.
2. Retries amplify the load. Every 429 invites a retry, and if the client, the orchestrator and the gateway each retry, the provider receives more traffic exactly when it can least absorb it. Our article on who owns the retry explains how retry storms form and why one layer must own the retry.
3. Orchestrator processor (CPU) and the event loop. Raise the model quota and the next limit is often your own compute: assembling prompts, serialising product candidates, counting tokens and streaming. One blocking call inside an asynchronous worker can stall every journey it serves.
4. Database waits and locks. Each turn writes session state and a basket draft. Under concurrency, writes queue, and the basket agent and the app can contend for the same basket row.
5. Connection pools. Autoscaling adds orchestrator pods, and each pod opens its own pool of database connections. Pods multiplied by pool size can exceed what the database allows, so the fix for step 3 creates step 5. Part 12 covers this hidden architecture.
6. Cache evictions and misses. By Little's law, from Part 7, in-flight work equals arrival rate times time in system. As journeys slow, more are live at once, so the cache holds more sessions and distinct lookups. Evictions rise, hit rates fall, more requests reach search and the catalogue, those slow down, and journeys stay live longer still. That feedback loop is why cache pressure often arrives late and fast.
Fixing the wrong constraint
Ubuntu Grocer's first instinct was the obvious one. The gateway showed 429s, so Ruan Botha, the AI engineering lead, asked for more model quota. The 429s fell and orchestrator CPU saturated. Ayesha Patel's SRE team raised the autoscaling ceiling. Pods multiplied, connection errors appeared at the database, and for a day the incident was misread as a database performance problem. Each change was reasonable. Made together, without a retest between them, nobody could say which had helped and which had moved the problem.
One finding mattered more. As planning calls timed out, the orchestrator fell back to a simpler plan that, in that early build, ignored soft preferences. The vegetarian grandson still got a vegetarian dish only because the dietary rule was enforced in code, as Part 4 requires, not in the prompt. Load changes behaviour as well as speed. Test both.
Find the knee, then fix one constraint at a time
The method that worked is unglamorous and repeatable.
- Ramp in steps with plateaus. Use open arrivals for journeys and the same seed every time. Hold each step long enough for queues and caches to settle.
- Record utilisation, saturation and errors for every resource at every plateau: model deployments, gateway, orchestrator, database, connection pools, cache. Brendan Gregg's USE method is a good checklist.
- Locate the knee. It is the step at which journey p95 starts rising faster than load. The first resource showing saturation at that step is the current bottleneck.
- Change one thing, rerun the same seed, and log the result. A short bottleneck log, with the date, the change and the before-and-after knee, becomes the most useful capacity document the team owns.
- Choose an operating point. Plan to run each constrained resource well left of its knee, with headroom for the payday step. The margin depends on how bursty traffic is and how fast you can add capacity; write it down as a decision.
Then design for the moment the knee is crossed anyway. Admit journeys at the backend-for-frontend (BFF) rather than letting every step queue. Give basket-critical steps priority over extras such as recipe inspiration. Shed work early and degrade honestly, for example by showing the recipe and a non-personalised list, which Part 10 develops. Write journey targets as service level objectives with error budgets, so "how much slower is acceptable on a payday Sunday" is decided in advance, not during the incident.
Questions to take into your architecture review
- For each constrained resource, what is its utilisation at today's peak, and where is its knee?
- Is journey p95 measured end to end, or only per service?
- Over what window does each model provider and gateway enforce its limits, and has a burst test confirmed it?
- Which single layer owns retries, and what happens to traffic when the provider returns 429?
- What do pod count multiplied by pool size add up to, against the database connection limit, at the autoscaling maximum?
- How does cache hit rate change as in-flight journeys rise?
- Where is the bottleneck log, and was each change retested with the same seed?
- What does the system do when arrivals exceed capacity: queue, shed, degrade or fail?
Key points
- Latency grows with one over one minus utilisation: gently at first, then steeply past the knee. The tail grows faster in absolute terms.
- Journeys multiply the tail: sequential steps add, and parallel fan-out waits for the slowest branch.
- The bottleneck moves. A typical order is model quota, retries, orchestrator CPU, database waits, connection pools and cache pressure, and some fixes create the next limit.
- Find the knee with a stepped, seeded ramp, record utilisation, saturation and errors, and fix one constraint at a time.
- Load changes behaviour, not just speed. Check that fallbacks keep hard constraints under pressure.
CloudNala's point of view
The most expensive capacity mistakes we see are not under-provisioning. They are confident changes made without a retest, on a system whose bottleneck has already moved. One fraction and one law explain most peak surprises, and both fit on a whiteboard. A team that knows where its knee is, and has written down how far left of it the business wants to run, can approach Black Friday with a plan instead of a war room.
Next: Part 9, Tokens are a capacity dependency. All parts are on the series page.
How CloudNala can help
CloudNala runs capacity investigations for retail and service agents. We help your team run a stepped, seeded ramp against the full journey, locate the knee for each constrained resource, read the evidence across gateway, compute, database, pools and cache, and keep a bottleneck log, so the next seasonal peak is planned rather than survived.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za