On the Saturday before Christmas, a shopper is planning lunch for twelve: a vegetarian main for her sister, a cousin who keeps halaal, trifle, and a budget she has revised twice. By the seventh turn every message carries the conversation so far, the current plan, the household's constraints and dozens of candidate products with pack sizes and allergen attributes. Thousands of shoppers are doing the same that morning.
None of them sees a token. All of them depend on there being enough.
Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery. Its people, systems and figures are illustrative and do not describe any single organisation. Part 8 found that the first limit to bite under rising concurrency was often a model quota or rate limit. This part explains why, and gives Pieter van Wyk, Ubuntu Grocer's principal architect, a model to fill in before the festive season rather than after it. The outcome at risk is the busiest selling week of the year, and the capacity commitments made for it.
Tokens are a rate as well as a bill
The Economics of AI Delivery series treats tokens as the meter on the invoice and argues for measuring cost per accepted outcome. This part asks a different question of the same tokens: not what they cost over a month, but whether the platform can process them fast enough in the busiest minute.
Providers express capacity as rates. Microsoft's Azure OpenAI service assigns quota per model and deployment in tokens per minute (TPM), with a requests-per-minute (RPM) limit set in proportion. Anthropic's Claude API limits requests, input tokens and output tokens per minute separately for each model class. OpenAI publishes RPM and TPM limits, among others, per organisation and project. Each is a ceiling per model or model group, so a journey using a strong model and a smaller one has two sets of limits to fit.
The capacity model
The model is built up from a single call. Every variable can be estimated before launch and measured from traces after it.
- Input tokens per call = instructions + shopper profile and constraints + conversation history + tool results such as product candidates.
- Output tokens per call, counted separately: some providers limit them separately, and they drive latency.
- Calls per step, including the matcher's batches; retries sit in the peak factor below.
- Steps per turn and turns per journey, from the workflow in Part 6 and the conversation shapes in Part 7.
- Journeys started per minute at peak, from Part 7's arrival model.
In steady state, the token rate a deployment must sustain is journeys started per minute multiplied by that model's tokens per journey. Journey duration does not change the rate; it changes how many are in flight. Multiply by a factor for bursts and retries, divide by the utilisation you will run at, and compare with the limit.
How the limit counts matters
The same nominal limit can admit very different traffic, because providers count differently (checked on 5 November 2026):
- Reserved output. OpenAI counts the larger of
max_tokensand the estimated token count, so a generous output cap consumes limit you may never use (OpenAI). Anthropic saysmax_tokensdoes not factor into its output-token limit (Claude documentation), and Azure OpenAI counts actual input and generated tokens (Microsoft Learn). - Cached input. For most Claude models, tokens read from the prompt cache do not count towards the input-token limit. On Microsoft's provisioned throughput, cached tokens do not consume capacity, and output tokens weigh more than input tokens when sizing (Microsoft Learn). Caching can be a capacity lever on one platform and only a cost lever on another; see prompt caching is an architecture decision for the design side.
- Short windows. As Part 8 showed, per-minute limits are enforced over seconds, so size for the peak second, not the average minute.
Gateways are a second ceiling. Gateways such as Azure API Management, through its llm-token-limit policy, limit tokens per minute per key and returns HTTP 429 (too many requests) when exceeded; its documentation warns that concurrent requests can temporarily overshoot, and that each regional gateway counts independently (Microsoft Learn).
A worked example, with illustrative numbers
Every figure below is illustrative and round. Ubuntu Grocer's average journey has four turns. Each turn makes one planning call on the stronger model, and two matcher calls and one basket call on a smaller model. The planning call carries 3,000 fixed input tokens (1,500 of instructions, 500 of profile and constraints, 1,000 of plan state) plus 1,000 of history per earlier turn, and returns 600. The matcher and basket calls are stateless: each matcher call takes 4,000 input tokens of product candidates and returns 200; each basket call takes 2,500 and returns 300.
| Per journey (4 turns) | Stronger model | Smaller model |
|---|---|---|
| Input tokens | 4 × 3,000 + history (0 + 1,000 + 2,000 + 3,000) = 18,000 | 8 × 4,000 + 4 × 2,500 = 42,000 |
| Output tokens | 4 × 600 = 2,400 | 8 × 200 + 4 × 300 = 2,800 |
| Requests | 4 | 12 |
| At 300 journeys started per minute | 5.4 million input TPM, 720,000 output TPM, 1,200 RPM | 12.6 million input TPM, 840,000 output TPM, 3,600 RPM |
| Times 1.4 for bursts and retries, at 70% target utilisation | about 10.8 million input, 1.4 million output, 2,400 RPM | about 25 million input, 1.7 million output, 7,200 RPM |
The final row must fit, per model, inside provider limits, provisioned capacity and gateway quotas, in each location that serves traffic. Here, 1.4 divided by 0.7 means planning for twice the average demand.
Two levers show up at once. If the planning call's 2,000 tokens of instructions and profile are cached and the provider does not count cache reads, counted planning input falls from 18,000 to 12,000 per journey, since only the first turn writes the cache. And keeping the matcher stateless matters more than any trimming: it is the largest consumer, and its size depends on the shopping list, not the conversation.
History grows with the square of the conversation
The planning call's input rises by a fixed step every turn, so the total over a journey rises much faster.
If each turn adds h tokens of history and the whole conversation is resent, a journey of T turns carries h × T(T−1)/2 history tokens in total. Six turns carry 15h. Twelve turns carry 66h, more than four times as much for twice the conversation. Summaries flatten the curve; our articles on why conversation history is not a memory strategy and on context trimming without losing the evidence cover what to keep. For a retail agent, the shopper's hard constraints are evidence and must survive every summary.
Output tokens are latency
A step's duration is roughly time to first token plus output tokens divided by the model's output rate. With illustrative values of one second to the first token and 50 tokens per second, a 600-token meal plan takes about 13 seconds to finish. Streaming lets the shopper start reading sooner, but the matcher cannot start until it has the ingredient list. So ask the planning step for a compact, structured plan with the ingredient list first, and treat every output token as both latency and, on provisioned capacity, the more expensive kind of capacity.
Duration also sets concurrency. By Little's law, from Part 7, 1,200 planning calls a minute lasting 13 seconds each means about 260 in flight at once, and nearer 360 at the burst factor. Connection pools, gateway concurrency settings and client libraries must all allow that, or a queue forms before any token limit is reached.
The average journey is the wrong journey
Ubuntu Grocer's first capacity plan used the average journey and a December uplift on journey count. In the same illustrative terms, the Christmas lunch for twelve runs to ten turns and needs four matcher calls a turn for sixty ingredients. Its planning input is about 75,000 tokens, roughly four times the average journey's, and its matcher input is five times as large. Festive personas also arrive together.
A change in the persona mix can move capacity more than a change in journey count. Size by persona, using the scenario generator's weights for each season, and plan for the tail of tokens per journey, not the mean.
Turning the model into decisions
- Budget tokens per step, in code. The orchestrator caps input and output per step and fails a step cleanly rather than letting a prompt grow without limit.
- Separate limits by step and priority. Basket-critical steps get their own gateway key or deployment, so extras cannot starve them; virtual keys are the usual mechanism.
- Buy capacity on evidence. Provisioned capacity buys predictable throughput and latency, but pays off only once real traffic fills it; see reserved AI capacity explained. Some providers' provisioned offerings can spill overflow to a pay-per-use deployment, one way to cover bursts without sizing for the peak.
- Spread capacity across locations or providers, with fallback that keeps hard constraints, which Part 10 covers.
- Measure, then refill the model from traces, as Part 11 describes, and feed the results back into the scenario generator.
Questions to take into your architecture review
- What are input tokens, output tokens and calls per step for each agent step, and where were they measured?
- What are tokens per journey by persona, at the median and the tail?
- For each model deployment, what are peak input TPM, output TPM and RPM, including bursts and retries, against its limits?
- Does each provider count reserved output, cached input and short windows the same way your model assumes?
- Which gateway token limits apply, how are they keyed, and do regional gateways count separately?
- Is conversation history resent in full, and when is it summarised? What is protected from it?
- What is the output length of the planning step, and can downstream steps start before it finishes?
- How many model calls are in flight at peak, and do pools and gateway concurrency settings allow it?
Key points
- Tokens are a capacity dependency as well as a cost. Limits are per model deployment, as tokens and requests per minute, enforced over short windows.
- Build capacity from the single call: input and output tokens, calls per step, steps per turn, turns per journey and journeys per minute.
- Providers count differently: reserved output, cached input and output weighting change what a limit admits.
- History grows with the square of the conversation; stateless steps and summaries flatten it.
- Output tokens are latency, and latency sets concurrency. Size by persona and season, not by the average journey.
CloudNala's point of view
Most AI capacity plans we review are cost forecasts with a capacity label. They multiply an average by a volume and stop before the busiest minute or the heaviest persona. The variable-based model takes an afternoon to fill in with illustrative values and a few weeks of traces to make accurate, and turns arguments about quota into arithmetic. We would make it a release artefact for any agent that faces customers at peak, reviewed as seriously as database sizing.
Next: Part 10, Model routing without losing the customer. All parts are on the series page.
How CloudNala can help
CloudNala builds token capacity models for customer-facing agents. We help your team break each journey into steps and calls, fill the model with illustrative and then traced values, check it against how each provider and gateway counts, and decide where caching, stateless steps, summaries and provisioned capacity help most in the busiest minute of the year.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agentic Retail Review or write to us at consult@cloudnala.co.za