Load-test the shopping journey, not an endpoint

The load test report was green. A pool of virtual users had sent the same message to the meal assistant's chat endpoint, again and again: "supper for four tonight, under R400". Every response came back within target. Nobody had asked for a week of school lunches, mentioned a peanut allergy, changed their mind about the chicken, or stopped halfway through to answer a child. Nobody had shopped.

Two weeks later, on the Sunday after payday, real shoppers did all of those things at once.

Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on lessons from real enterprise retail-agent delivery. Its people, systems and figures are illustrative and do not describe any single organisation. Part 6 chose a workflow with bounded agent steps for its meal assistant. This part opens the production-engineering arc by asking what the green report could not: will that journey hold when thousands of different shoppers use it at once?

The outcome at risk is simple. A launch decision, a festive-season campaign and a capacity budget were about to rest on a test that measured almost nothing the shopper would feel.

An endpoint test measures the wrong unit

The unit of value at Ubuntu Grocer is a journey: a shopper's session from first message to a basket they accept. One journey is several turns. Each turn passes through the backend-for-frontend (BFF) to the journey orchestrator, which calls the intent and meal-planning agent, the ingredient-to-product matcher and the basket agent. Each of those makes one or more large language model (LLM) calls through the AI gateway, and the matcher fans out to search, pricing and availability for every ingredient.

A single repeated prompt flatters every one of those layers:

• Caches warm on the first request. The same prompt resolves to the same twelve products, so every lookup hits and the test measures cached performance real traffic will never see.
• Prompts stay short. A one-turn conversation has no history. Real shoppers refine, and every refinement carries the conversation so far.
• Fan-out stays small. "Supper for four" needs six or seven ingredients. "Meals for the week and a braai on Saturday" needs forty, each a search, a ranking call and an availability check.
• Nothing is missing. No out-of-stock items, so substitution never runs. No allergen, so the hard-constraint filter from Part 4 never forces a second search.
• Nobody leaves. Abandoned journeys leave sessions and basket drafts behind. A test where everyone finishes never accumulates that residue.

The deeper problem is that an endpoint test reports latency per request, while the shopper experiences time per journey and, above all, whether the basket at the end is valid.

The scenario generator

Ayesha Patel, Ubuntu Grocer's site reliability engineering (SRE) lead, replaced the single prompt with a scenario generator. It is a small piece of software, owned by the platform team, that writes one journey script per synthetic shopper by drawing from seven independent distributions. Because it is seeded, any run can be replayed exactly after a fix.

Three design choices make the generator useful rather than merely busy.

Draw from the real catalogue. Lists are generated against the live catalogue and real store availability, which pushes lookups into the long tail, where caches are cold and the data is weakest.

Make abandonment depend on latency. A fixed abandonment rate is decoration. Real shoppers leave more often when the plan is slow, and that feedback is what turns a slow step into lost revenue. Report abandoned journeys as a first-class result.

Never let a synthetic shopper buy anything. The agent proposes and the shopper confirms; nothing reaches checkout without explicit shopper action. Synthetic journeys stop at basket review, use test accounts and carry a tag that keeps them out of analytics, loyalty points and recommendations. Learn from production as distributions, not transcripts: replaying real conversations would copy allergy and religious dietary information, which the Protection of Personal Information Act (POPIA) can treat as special personal information (health, religious belief), into a test system; see Part 4. Our article on what POPIA means for your data and AI strategy covers the wider point.

Open and closed workload models

The arrival model decides what the test is capable of seeing.

In a closed model, a fixed pool of virtual users each sends a request, waits for the reply, thinks, and sends the next. If the system slows, the rate of new requests falls. The test backs off exactly when the system struggles, so the queue never builds. The documentation for the open-source load-testing tool k6 notes that in a closed model response time influences throughput, a problem "known as coordinated omission" in some testing literature, and that its arrival-rate executors implement the open model instead (k6 documentation, checked on 5 November 2026).

In an open model, new work arrives at a set rate regardless of how the system is coping. That is how journeys start: a push notification goes out at 15:00 and people open the app whether the assistant is fast or slow.

Little's law makes the difference concrete: the average number of items in a system equals the arrival rate multiplied by the average time each spends there (Little, 2011). Take an illustrative 20 new turns per second, each taking 4 seconds: about 80 are in flight. If a dependency slows and turns take 8 seconds, an open model shows about 160 in flight, double the sessions, connections and memory. A closed test with 80 virtual users and no think time shows the same 80 in flight and half the throughput. Only the open model shows the pile-up real shoppers would create.

Real shopping is partly open, the term Schroeder, Wierman and Harchol-Balter use in their paper showing how differently the two models can behave (NSDI 2006). For a retail agent that is the right default: open arrivals for journeys, and a closed loop with drawn think time inside each one.

Judge the run on journey outcomes

Ubuntu Grocer reports, for each step of the arrival ramp:

• Journey success rate: the share of journeys reaching a valid basket the shopper could accept.
• Time to first plan and time to basket, as percentiles. The 95th percentile (p95) is the time within which 95% of journeys complete.
• Tokens and model calls per journey, by persona, because averages hide the expensive tail.
• Rejections and fallbacks: HTTP 429 (too many requests) responses, retries, and every fallback to a simpler path.
• Abandonment by step.
• Hard-constraint violations, with a target of zero.

The last measure is the non-obvious one. Under load, timeouts trigger fallbacks, and a careless fallback can drop the allergen filter to save a call. A test that measures only latency will pass a system that has quietly started proposing baskets with peanuts in them. Each synthetic journey carries a correlation ID, as Part 11 describes. Write the journey targets as service level objectives before the test, not after it.

What the rebuilt test found

The first generator run did not fail where the team expected. Compute was fine. The first limit to bite was the model quota on the planning step, and only for long, multi-turn personas whose prompts had grown with every refinement. The matcher's cache hit rate fell sharply once lists spread across the long tail. The cache was keyed on the ingredient phrase and store, so a popular "supper for four" shared entries across thousands of shoppers, while long-tail items, local-language terms and misspellings produced keys seen once and evicted before reuse. The hit rate was a property of the traffic mix, not of the cache. Abandoned journeys left basket drafts nobody cleaned up. None of this could appear in the single-prompt test. Part 8 follows what happened as concurrency rose further.

The trade-offs

Synthetic shoppers spend real tokens. A capacity run against real model endpoints is metered like production, so budget it like a pilot; our guide to quoting AI run cost applies. Run most tests against a stub model endpoint that returns realistic latency and token counts, and reserve real-model runs for capacity and quality. Stubs cannot reveal provider limits, so they never replace the real run.

Shared quota can hurt real shoppers. A test drawing on production model quota can throttle live customers. Give tests their own deployment or gateway key with its own limits; virtual keys are the usual mechanism.

The generator needs an owner. Distributions go stale. Recalibrate them from production traces every quarter and before each seasonal peak.

Questions to take into your architecture review

1. What is the unit of your load test: a request, a turn or a complete journey ending in a basket?
2. Which personas, constraint mixes and conversation lengths does the generator draw from, and where did their weights come from?
3. Are ingredient lists drawn from the live catalogue and real store availability, including the long tail and out-of-stock items?
4. Is the arrival model open for journeys, with think time drawn from a distribution inside each journey?
5. Does abandonment depend on latency, and are abandoned sessions and basket drafts measured?
6. Can any run be replayed exactly from its seed after a fix?
7. Does every synthetic basket get a hard-constraint check, and what is the tolerance for a violation?
8. How is synthetic traffic tagged, kept out of checkout, analytics and loyalty, and kept off production quota?

Key points

• The unit of a load test is the journey, not the endpoint. A single repeated prompt warms every cache and hides fan-out and history growth.
• Build a seeded scenario generator over personas, lists and constraints, conversation shape, think time, arrival rate, abandonment and cache variety.
• Use an open arrival model for journeys and a closed loop with think time inside them. Closed-only tests hide the queue.
• Judge each run on journey success, time to basket, tokens per journey, fallbacks, abandonment and hard-constraint violations.
• Keep synthetic shoppers out of checkout, analytics and production quota, and learn from production as distributions, not transcripts.

CloudNala's point of view

Most retail-agent load tests we review are performance tests of a chat endpoint, and they answer a question nobody asked. The test that matters looks like a payday Sunday: varied households, long conversations, cold caches and shoppers who leave when it is slow. A seeded scenario generator is a few weeks of work and becomes a permanent asset, rerun before every peak and model change. We would rather see a modest system tested honestly than an impressive one tested with a single prompt.

Next: Part 8, What thousands of concurrent shoppers teach you. All parts are on the series page.

How CloudNala can help

CloudNala designs journey-level load tests for retail and service agents. We help your team build a seeded scenario generator from your own catalogue and traffic, choose an arrival model that matches how customers arrive, separate test quota from production, and define the journey outcomes that decide whether a release is ready for its first seasonal peak.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agentic Retail Review or write to us at consult@cloudnala.co.za