An AI agent can complete a task, produce a polished answer and still leave the organisation unable to answer basic questions. What did it actually do? Which model, tools, memory and data did it use? Which steps failed or were quietly skipped? Was the answer good enough? What did the complete task cost? Was the agent authorised to do everything it attempted? And who has to investigate, approve or correct the outcome?
If those questions cannot be answered, the organisation has an impressive demonstration, not an operable service.
This series is about the capability that answers them, which we call the agent evidence layer: the telemetry, evaluation, policy, cost and accountability capabilities needed to understand and govern an agent after the prompt is sent. It is written for the people who approve agents, build them and carry the risk when they go wrong, and it starts where most executive conversations about agents start: with the bill.
The camera, the referee, the receipt and the rules
The easiest way to hold the distinction in your head:
Observability is the camera. Evaluation is the referee. Metering is the receipt. Governance defines the rules, authority and consequences.
That framing comes from a useful public walkthrough. Sean Chen's video "You Can Learn AI Agent Observability & Eval In 20 Min", published on his channel Sean's AI Stories on 6 October 2026, builds a small research agent and makes its behaviour visible step by step. The agent's model, tool and memory calls appear as parts of a trace. Tool calls go through treg, a tool gateway that describes itself as "OpenRouter for agent tools": one credential for thousands of tool endpoints, paid per call. The loop, memory, evaluation and per-turn traces run in Waku, which describes itself as portable memory for AI agents ("one store, every harness") and ships with an open-source, local-first agent harness. Chen founded AutoManus, the company behind Waku, so treat the walkthrough as a builder's demonstration rather than an independent review.
The demonstration's own numbers are a good place to start. One research run cost 29 US cents. A later run cost 54 cents, of which 53 went to the model. Those are figures from one demonstration, not benchmarks, and nobody should budget from them. What the walkthrough does well is make cost and behaviour visible at all, which is further than many production pilots get.
The analogy is a beginner's model, and production needs more precision than it gives. A camera can record an incident without deciding whether the behaviour was acceptable. A referee can make a call without knowing what the match cost. A receipt can show what was charged without proving the purchase was authorised. Production agents need all four capabilities, connected by a common evidence model. The seven parts of this series take them in turn:
- Your agent spent the money. Can you prove why? Can every rand be connected to a useful outcome? (this part)
- A trace is not an evaluation: separating execution evidence from quality judgement.
- When the tool fails but the answer looks complete: exposing incomplete evidence and partial failure.
- Memory is a governed data product: remembering safely, accurately and with permission.
- The gateway is a control point, not a trust boundary: controlling tool access without creating a new single point of failure.
- Evals that gate real work: what evidence is strong enough to release an answer or an action.
- From agent dashboard to operating model: who owns the agent once it is live.
Meet Weaverbird Agri
Weaverbird Agri is fictional, and its systems and figures are illustrative.
Weaverbird is an agricultural inputs distributor headquartered in Bloemfontein. It sells seed, fertiliser, crop protection and animal health products to commercial farmers and co-operative dealers through about 38 depots across the Free State, the Northern Cape and North West, and employs around 650 people. It runs two agents in production, looked after by a four-person digital desk and the group IT platform team.
- Scout is a market-research agent for the commercial team. Each week it produces commodity and competitor briefs: fertiliser import prices, the rand/dollar exchange rate, the maize and soybean outlook, competitor promotions. It plans with a small model, searches several sources, reads an internal price history store and its memory of earlier briefs, writes with a large model and checks its citations with an evaluation model. A person reviews every brief before it goes out. It produces about 30 briefs a month.
- Depot Desk answers dealers' order, stock and delivery questions in the dealer portal and on WhatsApp. It reads the ERP and can draft credit notes for short deliveries. A depot manager approves credit notes up to R5,000 with one click, and anything larger goes to the credit controller. It can never issue a payment or change a customer's credit limit.
A gateway sits in front of both agents for metering, policy and the tool catalogue. Not everything goes through it, and that turns out to matter.
The bill is the end of the story, not the explanation
In August the CFO asked a simple question. AI spend had risen by roughly 40% month on month, and Scout had produced the same number of briefs. Why?
The dashboard could show the monthly total and a breakdown by model. It could not explain the rise. That is the gap between a cost number and cost governance. Four questions sit behind every agent bill:
- Does the number include the entire task?
- Did the task achieve anything worth paying for?
- Can the cost be allocated to the right customer, product, department or business process?
- Did the agent stay within its approved budget and authority?
Agent costs behave differently from those of an ordinary API. One request can start a loop of unknown length. The agent may pick different models, repeat searches, pull large documents into context, call paid tools and ask another agent to review its work. A retry policy can turn one failure into five paid attempts. A long context window can make the final synthesis more expensive than every earlier step combined. The organisation therefore needs a cost receipt for each meaningful unit of work, not just a monthly total.
What belongs on the agent cost receipt
| Cost component | Evidence to capture | Common blind spot |
|---|---|---|
| Model inference | Provider, model, deployment, input, output and cached tokens, unit price, currency | Price changes, or a silent fallback to a different model |
| Tool execution | Tool, operation, provider, charge, result status, retries | A free trial or zero-priced call treated as permanently free |
| Retrieval and memory | Reads, writes, embeddings, storage, index queries | Vector and database operations left out |
| Orchestration | Compute time, queueing, workflow engine, serverless calls | The agent host treated as costless plumbing |
| Data movement | Network traffic, egress, document transfer | Cross-region retrieval and large payloads |
| Evaluation | Model judges, test execution, human review | Quality assurance left out of unit economics |
| Security and operations | Secrets, logging, monitoring, retention, incident handling | Shared platform costs never allocated |
| Human intervention | Review, approval, correction and exception handling | The automation claim ignores the manual work |
Here is one Scout brief, itemised:
The model and tool bill for that brief is R7.60. The approved brief costs R103.60, because the commercial analyst who reviews it spends eight minutes correcting an unsupported claim. Both numbers are true; they answer different questions. If the reviewer keeps correcting the same kind of failure, that is evidence for an engineering change. If the correction cannot safely be automated, it is part of the operating cost, and the business case has to carry it.
Notice the second line: eight search calls, two of them failed. They were paid for. Failed and retried work is a cost line in its own right, and it is the first place to look when spend rises without output rising.
Cost must be read at more than one level
Per-token prices are raw ingredients. Leaders need several views of the same cost evidence, each answering a wider question.
- Run. What did this specific run cost? Which step consumed the most? How much was spent on failures, retries or duplicate work?
- Task. A business task can contain several runs: one agent researches, another drafts, a third reviews and a person approves. The task cost is the sum of the workflow, not the first trace.
- Product and customer. Which product, tenant, customer or cost centre created the demand? Shared infrastructure needs an allocation rule that can be explained and repeated. Scout's cost belongs to the commercial team; Depot Desk's belongs to depot operations.
- Outcome. What did the spend buy? A R20 task that resolves a dealer's case can be better value than a R2 task that produces a plausible but unusable answer.
- Portfolio. Which use cases are scaling economically? Which agents cost more each month without improving resolution time, conversion, service quality or risk?
The engineer lives at the first level and the CFO at the last two. The evidence has to connect them, or the conversation between them goes nowhere.
The gateway does not see everything
A model or tool gateway is a valuable metering point because it records the requests that pass through it. It is not a complete ledger by default. Costs bypass it when:
- an application calls a provider's SDK directly;
- a background process carries on after the user's request ends;
- a remote MCP server does work behind its own boundary;
- a vector database, search engine or document processor is billed separately;
- network egress or cross-region processing is charged outside the agent platform;
- a person reviews, rewrites or approves the result;
- a provider applies credits, discounts or tiered pricing later.
Weaverbird found three of these at once. When the digital desk added up the gateway's monthly total, it did not match the provider invoices. The gap was the vector store, which is billed separately; egress charges; and the ERP connector calls that Depot Desk's orchestrator makes directly, without going through the gateway.
So the internal receipt has to be reconciled against provider billing. When the numbers differ, the team should be able to say whether the cause is delayed billing, currency conversion, uninstrumented calls, shared services or a wrong unit price. For a South African organisation, currency belongs on that list in its own right: most model and tool pricing is in US dollars, budgets are in rand, and the exchange rate moves between the run and the invoice. Record the price in the currency in which it was charged, along with the rate used for the rand figure.
Budget controls must act before and during the run
A dashboard that reports overspend tomorrow is useful for analysis. It does not protect today's budget. Controls that do:
- maximum spend per run, task, customer and day;
- maximum agent iterations and tool calls;
- token and context limits;
- approved model tiers by task type;
- cheaper model routing for classification and extraction;
- escalation before an expensive search or premium data source is used;
- circuit breakers when cost, latency or error rates cross a threshold;
- idempotency controls, so a retry cannot repeat a paid action.
These are not one-size-fits-all. A R5 ceiling may be sensible for summarising an internal memo and dangerous for a due-diligence task, where incomplete research creates a far larger risk than the spend it saves. Weaverbird set Scout's ceiling at R50 of model and tool spend per brief before a person has to approve going further, which is well above its normal R7.60 and low enough to stop a runaway loop.
What the CFO's question came down to
With receipts in place, the August rise explained itself in an afternoon. Two changes accounted for almost all of it.
First, one of Scout's search providers had started rate-limiting requests. Scout's retry policy kept trying, and every retry was a paid call. The number of briefs was flat; the number of attempts per brief was not.
Second, someone had improved Scout's prompts by including two years of price history in every request, so the model would "have the context". Synthesis calls became much larger. The briefs were no better: a reviewer could not point to anything the extra history had added.
Neither change was visible in the monthly total. Both were obvious on the receipt. The fixes were cheap: a retry limit with a recorded limitation (the subject of Part 3), and retrieval of the relevant price window rather than all of it. The first change also showed a quality problem hiding behind the cost problem: some of those retries eventually gave up, and the brief did not say so.
Existing articles cover the mechanics of cost in more depth: why production AI must be visible, and the developer, team and finance views of AI cost.
Questions leaders should ask
- What is the cost unit: token, run, task, case, customer or outcome?
- Which costs are measured, which are estimated and which are left out?
- Can every provider charge be reconciled to a trace or an allocation rule?
- How much spend comes from failure, retry and rework?
- What budget stops the agent before the damage grows?
- Which business measure tells us whether the cost was worthwhile?
- Who is accountable when the agent spends correctly and achieves nothing?
The short version
Put a receipt on every task. Include model, tool, memory, platform, evaluation and human effort, not only tokens.
Read cost at five levels. Run, task, customer, outcome and portfolio answer different people.
Reconcile against invoices. A gateway sees what passes through it, and some costs don't.
Control before and during the run. Limits, circuit breakers and idempotency protect today's budget; dashboards explain yesterday's.
Measure value, not cheapness. The aim isn't the cheapest possible agent. It's a controlled cost for an outcome the organisation can verify.
Next: Part 2, a trace is not an evaluation. See every part on the series page.
How CloudNala can help
We help organisations put a cost receipt on their agents before the CFO asks for one: defining the cost unit, instrumenting model, tool, memory and platform calls, reconciling internal measurements against provider invoices, setting budgets and circuit breakers that act during the run, and building cost-per-outcome reporting that an executive can actually use. We do the same work in pilot reviews, so the business case includes the human review and platform costs that demonstrations leave out.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agent Evidence Review or write to us at consult@cloudnala.co.za