A trace is not an evaluation

An agent trace can show that the model received a prompt, called a search tool, retrieved three memories and produced an answer in 42 seconds. None of that proves the answer was correct.

Teams get into trouble when they use telemetry, observability, evaluation and audit as if they meant the same thing. They do not. A trace is execution evidence: a record of what happened. An evaluation is a judgement: whether what happened was good enough against a stated standard. Mixing them up produces two familiar failures: dashboards full of green lights over answers nobody has checked, and quality reviews that cannot say which step went wrong.

Part 1 put a receipt on the agent's work. This part is about the camera, and about where the camera stops being enough.

Eight words that are not synonyms

The first four are telemetry. Metering is telemetry put to financial use. Evaluation, audit evidence and business outcome are something else again: they need criteria, retention rules and owners that telemetry alone does not supply.

A standard vocabulary, still being written

OpenTelemetry is the vendor-neutral framework most platforms now use for traces, metrics and logs. Its semantic conventions for generative AI give names to the things an agent does, so that a trace from one framework can be read by another tool without a translation layer. In June 2026 they moved into their own repository, open-telemetry/semantic-conventions-genai, and every document in it is still marked Development, which means not yet stable.

Even in development, the conventions describe the shape of an agent run well. There are agent spans for create_agent, invoke_agent, invoke_workflow and execute_tool; operation names for chat, embeddings, retrieval, planning and memory operations such as search_memory and update_memory; coverage for MCP, where a tool call is recorded with gen_ai.operation.name set to execute_tool; and usage attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. One detail matters for anyone building the cost receipt from Part 1: the input token count includes cached tokens, which are usually billed at a different rate, so the receipt has to read the cache attributes too or it will misprice the run.

The practical advice follows from the status. Use the conventions rather than inventing your own field names, pin the version you implement, and review it when the conventions change. Treating an evolving standard as settled is how a dashboard quietly stops matching the data underneath it.

What a useful agent trace contains

A trace identifier on its own has little value. Each span needs enough structured context to support diagnosis without indiscriminately storing sensitive content. A useful set of fields:

• trace, run, session and task identifiers;
• parent and child span relationships;
• agent, workflow and software version;
• model provider, model and deployment identifier;
• tool name, operation and version;
• start time, end time and duration;
• request, response and error status;
• token and usage counts;
• cost estimate and currency;
• retry count and reason;
• policy decision and rule version;
• user, tenant and delegated identity references;
• data classification and processing region;
• evidence and citation identifiers;
• outcome and evaluation references.

Weaverbird Agri is fictional, and its systems and figures are illustrative.

Here is one Scout run from Weaverbird Agri, the fictional distributor this series follows, laid out as a waterfall of parent and child spans:

The parent-child relationships are what make this useful. Without them the team sees a list of isolated API calls and cannot tell which task, user or final answer each one influenced. With them, anyone can see that the social media search failed with a rate-limit error after its retries ran out, that synthesis started after the price search finished, and that the run took 42 seconds.

What the trace cannot see is whether the brief Scout produced was any good. The evaluation span at the end did run, and it passed: every citation in the brief resolved to a document Scout had actually retrieved. That check was real and useful. It was also the wrong check for what had gone wrong, because the brief went on to say that no competitor promotions had been found on social media, and the evaluation had no rule about sources that were never searched. The trace held the evidence of the failure; the evaluation was not asking the question that would have found it. Part 3 follows that brief to its consequences.

Content capture is a data decision

Full prompts, retrieved documents and tool responses can contain personal information, confidential commercial data, credentials, or instructions planted by an attacker. Depot Desk's traces, for example, include dealers' names, account numbers and order history. Under POPIA, Weaverbird is the responsible party for that personal information wherever it lands, including in an observability platform, and a hosted tracing provider is an operator processing it on Weaverbird's behalf.

So content capture should be deliberate, classified, redacted and access-controlled. In many cases hashes, references, structured attributes and carefully sampled payloads are safer than recording everything. The test is whether the team can still diagnose a failure; it usually can without the dealer's full conversation in the span. For more on prompts as retained evidence, see the prompt is part of the evidence.

Observability needs questions, not just data

Telemetry becomes observability when the organisation can use it to understand what is happening inside the system. That means starting with the questions:

• Why did latency increase?
• Which model or tool caused the failure?
• Did the agent retrieve relevant context before answering?
• Which tenants are approaching their budget limits?
• Did a new prompt version increase retries?
• Are errors concentrated in one provider or region?
• Did the agent act after policy denied the request?
• Are successful-looking responses associated with poor business outcomes?

The last question is the one most dashboards cannot answer, because it requires joining execution evidence to what happened afterwards. Dashboards should be designed around decisions like these. A wall of charts that no owner uses is monitoring theatre.

Four views of the same evidence

The same trace identifiers serve four audiences, and each needs a different view.

• Operations: latency, availability, error rates, saturation, queues and dependencies. Depot Desk's service owner watches response times by channel, because WhatsApp and the dealer portal behave differently.
• Quality: task success, groundedness, completeness, safety, policy compliance and how often people override the agent.
• Financial: tokens, model and tool costs, budget consumption, unit economics and cost allocation, as in Part 1.
• Governance: identity, authority, data classification, consent, policy decisions, approvals, changes and retained evidence. When a dealer disputes a credit note, credit control needs to see who drafted it, who approved it and under which rule.

An engineer needs the failed span. A CFO needs cost per approved brief. A risk officer needs the policy and approval record. A product owner needs evidence that the workflow actually helps dealers. None of them is served by the other's dashboard, and all of them depend on the same identifiers being present and consistent.

Sampling and retention

Storing every detail forever is neither affordable nor responsible. A practical policy keeps:

• aggregated metrics for long-term trends;
• full traces for errors, safety events and consequential decisions, such as every credit note Depot Desk drafts;
• a sample of successful traces for performance and quality analysis;
• redacted payloads where content is necessary for investigation;
• audit evidence for as long as legal, regulatory and contractual requirements say;
• shorter retention for development experiments that use synthetic data.

The policy should also document what was not captured. The absence of telemetry is not proof that an event did not happen. If the sampling rule drops 90% of successful runs, an investigator needs to know that before concluding that a problem never occurred.

The minimum viable evidence contract

Before building a dashboard, agree a contract for what every run must record. A starting point:

run:
  trace_id: required
  task_id: required
  tenant_id: required
  agent_version: required
  policy_version: required
  identity_reference: required
  start_time: required
  end_time: required
  outcome_status: required
  model_usage: required
  tool_usage: required
  estimated_cost: required
  evidence_references: required_for_research
  evaluation_result: required_before_release
  approval_reference: required_for_high_impact_action

The exact fields will differ between organisations and agents. What matters is that the contract is written down, versioned and stable enough for teams and tools to rely on. Weaverbird applies one contract to both agents: Scout must carry evidence references because it does research; Depot Desk must carry an approval reference on every credit note because it proposes financial actions.

The contract is also where trace and evaluation meet. evaluation_result: required_before_release says that a run cannot be released without a judgement, and that judgement has to be recorded against the run that produced it.

The short version

A trace records; an evaluation judges. Keep the words, the systems and the owners distinct.

Use the shared vocabulary, carefully. OpenTelemetry's generative AI conventions are the right starting point; they are still in development, so pin the version you implement and review it.

Capture content on purpose. Traces hold personal and confidential data, and POPIA applies there too.

Design views for decisions. Operations, quality, finance and governance each need their own view of the same identifiers.

Write the evidence contract first. Then build dashboards on top of it.

A trace tells you what the system recorded. An evaluation tells you what that evidence means against a defined standard.

Previous: Part 1, your agent spent the money. Next: Part 3, when the tool fails but the answer looks complete. See every part on the series page. For the basics of evaluating agents, see how do we know an AI agent is working?

How CloudNala can help

We design agent telemetry that people can actually use: an evidence contract for every run, instrumentation based on OpenTelemetry's generative AI conventions with the version pinned, content-capture and redaction rules agreed with your privacy owner, sampling and retention policies, and dashboards built around the decisions each owner has to make. We keep traces and evaluations separate in the design, so a green dashboard means something.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agent Evidence Review or write to us at consult@cloudnala.co.za