Trace one basket across every dependency

A Sunday afternoon complaint reaches the support desk: "Your meal thing took forever, and then it put rice in my basket after I told it we already have rice." The shopper had asked for supper for four, with a peanut allergy in the house, under R400. She swapped the chicken for beef halfway through, removed the rice herself and checked out. She will not use the assistant again unless it is quicker and listens.

All the support desk has is her email address and "around three o'clock". The journey touched the app, the backend-for-frontend (BFF), the orchestrator, an AI gateway, several model calls, the catalogue, pricing and availability, a cache, a database, a queue and the basket service, each with its own logs and identifiers.

Ubuntu Grocer is a composite: a fictional South African grocery retailer whose scenario draws on real enterprise retail-agent delivery lessons. Its people, systems and figures are illustrative and do not describe any single organisation.

Part 10 designed routing, fallback and degradation for the journey. Those decisions are invisible unless they are recorded against the journey they affected. This part covers one correlation ID carried through every dependency, and the smallest operational view that answers a complaint like this one.

A journey is not a request

Distributed tracing records one request as a tree of spans, a parent for the request and children for each call it makes. A shopping journey is not one request. This one was four: the first planning turn, the swap from chicken to beef, the shopper pressing "add to basket", and the asynchronous basket write that followed through the queue.

So Ubuntu Grocer uses two identifiers with different jobs.

• The trace ID identifies one request and everything it caused.
• The journey ID identifies the shopper's conversation from first message to basket outcome. The BFF creates it once, when the conversation starts, because the BFF already owns the shopper's session. Every span in every trace of that journey carries it as an attribute, under a namespaced key such as ubuntu.journey.id.

The journey ID is pseudonymous, never the customer, email or loyalty number. Support finds it from the session record under access control; engineers never need to know who the shopper is.

Carry the context through every hop

For the trace ID itself, use the standard. The W3C Trace Context Recommendation defines a traceparent header that carries a version, a 16-byte trace ID, the 8-byte ID of the calling span and a set of flags, of which the only one defined in the current Recommendation is "sampled". A companion tracestate header carries vendor-specific detail. OpenTelemetry, the vendor-neutral observability framework, uses W3C Trace Context as its default propagator.

The standard is the easy part. The work is making sure no hop drops it, and in agent systems three hops often do.

The AI gateway. Gateways often terminate the incoming request and open a fresh one to the model endpoint. If they do not forward the context and record their own span, the trace stops exactly where the tokens, routing decisions and breaker events live. Ruan Botha's team made "gateway span present, with serving endpoint and location" an acceptance test for every route.

The queue. HTTP headers do not survive a message queue on their own. The producer writes the trace context into the message's metadata, and the consumer starts its own trace with a span link back to the producing span, so the basket write can be found from the turn that caused it.

The cache and the database. Without client instrumentation they appear as unexplained gaps in the parent span, which is where a slow availability lookup hides.

Here is the complaint's first planning turn, found by Ayesha Patel's site reliability engineering (SRE) team through the journey ID.

Two answers came straight off the screen. The strong model took longest, as expected; the surprise was the availability and price lookup, where a cache miss sent the read to the database because that Sunday's price changes had invalidated much of the cache at once. And the breaker event showed the gateway had moved the planning call to its second location, correctly.

The rice took one more click. The span attributes on turn 1 showed one pantry exclusion passed to the planning step. On turn 2, the swap from chicken to beef, the orchestrator had sent only the new message and the existing plan, and the planning span showed zero pantry exclusions. The model had done what it was asked; the defect was the orchestrator's state handling, fixed with a one-line change and a regression test, as every incident becomes a test case recommends.

Name model calls in a shared vocabulary

Model calls need consistent attribute names. OpenTelemetry's semantic conventions for generative AI provide them: an operation name, the requested and responding model, the provider, and token usage as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus agent spans such as invoke_agent and execute_tool. When we checked on 5 November 2026, the conventions lived in their own repository and their documents were still marked Development, which means not yet stable.

The practical advice from a trace is not an evaluation stands: use the conventions rather than inventing names, pin the version you implement, and review it when the conventions change.

The minimum useful operational view

A trace answers "what happened to this request?"; operators also need "what is happening to journeys?". Ubuntu Grocer's minimum view has four panels, joined on the journey ID.

1. Journey timeline. Every turn in order, with the gaps between them.
2. Per-step latency and tokens. For each agent step: duration, model tier, location, input and output tokens. This is where the token arithmetic from Part 9 meets reality.
3. Fallback, breaker and error events. Which rung of the degradation ladder each journey reached, which breakers were open, which outputs failed their quality checks, and how many products the constraint filter removed.
4. Basket outcome. Proposed, accepted as is, edited, abandoned, or carried to checkout by the shopper. This is the panel that connects engineering to the business outcome.

The fourth panel is the one most teams leave out. A journey that is fast, error-free and ends with the shopper deleting half the basket has failed, and without the outcome, tracing reports a healthy system while the product is not. The three views of AI cost observability and service level objectives (SLOs) and error budgets both get sharper when the unit is a completed basket rather than a request.

Sample for the journeys you will need

Keeping every span is expensive at payday volumes. Head sampling decides at the start of a request and sets the sampled flag so downstream services agree, but it decides before anyone knows whether the journey will go wrong.

Tail sampling decides after the trace is complete. The OpenTelemetry Collector's tail sampling processor, marked beta when we checked, keeps traces by status, latency and attributes, but it decides per trace, after a short wait (30 seconds by default), and needs every span of a trace on the same collector. It cannot see that turn 1 and turn 4 belong to one journey.

Keeping whole journeys needs extra machinery, and Ubuntu Grocer uses two pieces. The random share is decided deterministically from a hash of the journey ID, so every turn of a sampled journey gets the same decision. For journeys with an error, fallback, degraded rung, constraint-filter removal or slow turn, all spans first land briefly in low-cost storage, and a job promotes the whole journey once any turn qualifies. The alternative, routing spans by journey ID to collectors with a decision window longer than a journey, works too, at the cost of holding minutes of spans in memory.

Keep the shopper's allergy out of the trace

A trace that captures prompts and responses will capture "my husband is allergic to peanuts". Health information is special personal information under section 26 of POPIA, the Protection of Personal Information Act. This is general information, not legal advice; Ubuntu Grocer's information officer decides what is lawful.

The GenAI conventions treat message content as opt-in, and Ubuntu Grocer leaves it off in production. Spans record that a constraint was applied, its type and how many products it removed, not the shopper's words. Thandeka Mokoena, the security architect, approves any time-limited content capture for debugging. Our article on whether your observability tools store the prompt covers the copies that are easy to miss. OpenTelemetry also advises keeping personal information out of baggage, its mechanism for carrying key-value pairs between services.

Questions to take into your architecture review

1. Where is the journey ID created, and can you show it on every span of every trace in one journey?
2. Does your AI gateway forward traceparent and record its own span with the serving endpoint and location?
3. How does trace context cross your message queue, and are asynchronous steps linked back to the turn that caused them?
4. Which GenAI semantic conventions version have you implemented, and who reviews it when the conventions change?
5. Can an operator see, for one journey, the rung it reached, the breakers that were open and the products the constraint filter removed?
6. Is basket outcome recorded against the journey, and does any dashboard show it next to latency?
7. Does your sampling keep every journey with an error, fallback or degraded rung?
8. Where could a shopper's allergy or religious requirement appear in telemetry, and who approves capturing content?

Key points

• A journey is several requests, so it is several traces. Create a pseudonymous journey ID at the BFF and put it on every span.
• Use W3C Trace Context between services, and test the hops that usually drop it: the AI gateway, the queue, the cache and the database.
• Use OpenTelemetry's GenAI conventions for model calls, pinned to a version; they are still in Development.
• The minimum view is journey timeline, per-step latency and tokens, fallback and error events, and basket outcome.
• Sample whole journeys, and keep shoppers' health and religious data out of traces by default.

CloudNala's point of view

Most agent tracing we review stops at the gateway or starts again after it, hiding the most expensive, least predictable part of the journey. The fix is not a new tool but a correlation ID with an owner, a propagation test on every hop, and a basket outcome recorded next to the latency. Teams that add the outcome panel stop arguing about whether the agent is working, because the answer is on the screen.

Next: Part 12, Readiness, connection pools and the hidden architecture. All parts are on the series page.

How CloudNala can help

CloudNala designs end-to-end tracing for agentic journeys. We define the journey ID and its owner, test propagation through your gateway, queues and data stores, map model calls to the GenAI conventions, design journey-level sampling, agree content capture with your privacy owner and build the four-panel view.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agentic Retail Review or write to us at consult@cloudnala.co.za