The Agent Evidence Layer
An AI agent can finish a task, write a polished answer and still leave the organisation unable to say what it did, what it cost, whether it was good enough or who must act when it goes wrong. This seven-part series covers the evidence layer production agents need: the cost receipt, traces that are not mistaken for evaluations, failures that the final answer hides, memory as governed data, tool gateways as control points, evaluations that actually gate work, and the operating model that owns the result. One fictional South African agricultural distributor carries the examples.
7 articles
- Part 1Your agent spent the money. Can you prove why?An AI agent can finish a task, produce a polished answer and still leave nobody able to say what it did, whether it was good enough or what the whole task cost. Part 1 introduces the agent evidence layer and starts with the receipt: what belongs on it, the five levels at which cost must be read, what a gateway cannot see, and the budget controls that act before the bill arrives.13 min read
- Part 2A trace is not an evaluationA trace records what an agent did. It does not say whether the result was any good. Part 2 separates trace, span, metric, event, evaluation, metering and audit evidence, covers what a useful agent trace contains, where OpenTelemetry's generative AI conventions stand, the four views different owners need, sampling and retention, and a minimum evidence contract for every run.10 min read
- Part 3When the tool fails but the answer looks completeA language model can write fluently around a gap in its evidence, so a failed search turns into a confident finding. Part 3 covers the shapes partial failure takes, a research contract that defines required evidence before the agent runs, the failure-aware loop, why retries need boundaries, evaluations that test the claim rather than the error handler, and how to show uncertainty to the person reading the answer.9 min read
- Part 4Memory is a governed data productAgent memory is often described as a shared brain. In an enterprise it is stored, retrieved and inferred data, and it needs owners, provenance, permissions, quality controls and a way to forget. Part 4 covers the four kinds of memory, the memory lifecycle, what portable memory does to the threat model, memory poisoning, and how to test retrieval as part of memory quality.10 min read
- Part 5The gateway is a control point, not a trust boundaryA tool gateway gives agents one connection to many tools and gives the platform team one place for policy and metering. It also concentrates credentials, availability and authority. Part 5 covers the control plane and data plane, least authority, read and write tools, tool discovery as a policy decision, and what every consequential call should leave behind.10 min read
- Part 6Evals that gate real work"The answer looks good" is not a test plan. Part 6 covers five complementary forms of evaluation, an evaluation portfolio from offline testing to online monitoring, a worked scorecard, building a golden set from real failures, testing the whole system, and connecting every result to an action and an owner.9 min read
- Part 7From agent dashboard to operating modelEverybody can see the problem on the dashboard and nobody is sure who must act. The final part covers the production ownership map, the change record, agent incidents, balanced service objectives, a shared reference architecture for the evidence layer, a 90-day adoption path and the questions to ask before go-live.12 min read
Working through any of this in your own organisation?
These articles come out of delivery work rather than theory. If the problems in this series look familiar, a short scoping conversation is usually the quickest way to work out what is worth doing first.
Talk to CloudNala