The architecture is complete. Over nine chapters, Impala Bank's assistant grew from a public-content helper to a system that explains accounts, looks things up, takes bounded actions, opens cases, routes work across model paths and escalates through a stack of controls. The final element in the diagram is the one that keeps all the others honest: evidence.
Without it, the assistant's quality, cost and safety are matters of opinion. With it, the bank can answer four questions every month, with numbers: Is it still resolving clients' questions? Is it still safe? What does it cost per resolved question? And who is accountable when the answer to any of those is no?
Where we are in the case study
Impala Bank is fictional, and its systems and figures are illustrative. After Chapter 9, Ask Impala runs behind a named stack of input, retrieval, tool, policy, output and human controls, with business rules in a deterministic policy service and escalation designed as a control. In Chapter 1 the bank set its outcome, a repetitive question resolved with no repeat contact within seven days, against a baseline of 71%, and listed what must not get worse.
Traces: what happened, step by step
Every conversation produces a trace: the masked input, the classifier's decision, the model path chosen by the gateway, every retrieval and tool call with its parameters and result, every policy decision, the output checks, any escalation and the final response. Each step has a timestamp, a duration and, for model calls, a token count.
A trace answers "what did the assistant do?" It does not answer "was that good?" Our article A trace is not an evaluation makes the distinction in detail, and Impala built its practice around it. Traces are the raw material. Evaluations turn them into judgements.
Traces also inherit the boundaries from Chapter 4: signed-in traces are masked before storage, held separately, retained under the bank's records rules and readable only through controlled access.
Evaluations: was it good enough?
Impala ran evaluations at three speeds.
Before every change. A suite of about 1,200 test conversations, built from public conversations and synthetic clients, ran against every change to prompts, tools, policies, routing or models. Each test had an expected outcome: the right answer with the right source, the right tool with sensible parameters, the right policy decision, or an escalation. A change could not be released if it reduced the pass rate on any critical category, regardless of improvements elsewhere. Our evals that gate real work article describes this pattern.
Every week. A quality team scored a random sample of a few hundred real conversations against a written rubric: was the answer correct, complete, appropriately worded, and was the handoff right? Disagreements between scorers were discussed and used to sharpen the rubric. Every failure the weekly review found became a new test in the release suite.
Continuously. Automated checks on live traffic flagged patterns worth a person's attention: rising escalation for a question type, a tool failing more often, answers citing a page that has since been changed, a cluster of clients rephrasing the same question.
Cost per resolved question, not cost per token
The obvious cost measure, total spend on model tokens, was the least useful one. It rises when usage rises, which is the point of the programme, and it ignores the larger costs: the self-hosted classifier's GPUs, the platform team's time, content maintenance, the quality team and the contact centre's time on escalations.
Impala measured a fully loaded cost per resolved question, calculated the same way as the Chapter 1 baseline of about R38 so the two could be compared fairly. Tokens were one input among several. A conversation that used few tokens but ended in a repeat call counted as expensive, because the bank paid twice.
That measure changed some decisions. Shortening the system instructions saved a little on tokens. Fixing the twelve help pages behind most "no approved content" handoffs saved far more, by resolving questions that had been escalating. The AI observability and cost article covers the mechanics of controlling token spend; the lesson here is to measure the cost of an outcome, not of an input. The ZAR/USD exposure deserves a line in the budget too: hosted model prices are typically set in dollars, while the bank's budget is in rand.
The operating rhythm
Evidence is only useful if someone looks at it on a schedule and has the authority to act. Impala wrote down its operating rhythm and the owner of each beat.
Two lines at the bottom of that diagram mattered most in practice. The head of digital service, as product owner, is accountable for the outcome. The executive who approved the assistant's scope is accountable for its risk. The platform team runs the assistant but does not decide whether it should exist or what it may do. Separating those roles stopped a familiar drift, where the team that builds a system slowly becomes the team that decides its scope, because nobody else is in the room.
The quarterly review is the moment of real authority. Using the quarter's evidence, the risk committee decides whether the assistant keeps its current scope, gains a capability or loses one. In the illustrative first review after launch, Impala's committee kept the scope unchanged, asked for better vulnerability detection in Afrikaans and isiZulu conversations after the weekly reviews found misses, and deferred a proposal to add a fourth action until that work was done.
When to switch it off
Every production assistant needs a written answer to an uncomfortable question: under what conditions do we turn it off, partly or completely? Impala defined three. If any guardrail metric from Chapter 1 crosses its threshold, fraud linked to the channel, complaints or time for vulnerable clients to reach a person, the relevant capability is suspended pending review. If the release suite cannot be run, nothing is released. And the gateway can disable any model path, tool or action within minutes, routing affected conversations to people.
None of these switches has to be used for them to be valuable. Their existence changes how the scope conversation goes, because everyone knows a capability can be withdrawn as quickly as it was granted.
What the case study adds up to
Looking back across ten chapters, the architecture that grew was not built around a model. It was built around an outcome, a boundary, a set of narrow tools, a few bounded actions, existing workflows and people, a routing decision shaped by South African realities, a stack of controls and a body of evidence. The model is one box in the middle, and by Chapter 8 even that box became a gateway that can change its contents.
That is the general lesson. The demo shows that a model can answer a question. Production is everything that decides which questions it may answer, for whom, with what data, under whose authority, at what cost, and how the organisation will know when it has stopped working.
If you want to see the same principles from other angles, our Agentic Banking series looks at agents that act for clients across banks, and Building Governed AI Agents covers agent design and cost control more generally.
Key points
Traces record what happened; evaluations judge it. Keep both, and never mistake one for the other.
Evaluate at three speeds. Before every change, weekly by people, and continuously on live traffic.
Measure cost per resolved question. Compare it with the Chapter 1 baseline, fully loaded, not tokens alone.
Write down the rhythm and the owners. The product owner owns the outcome; the approving executive owns the risk.
Decide in advance when to switch it off. Guardrail metrics and fast kill switches keep scope honest.
That completes the case study. Start again from Chapter 1, Define the outcome before you build the assistant, or see all chapters on the series page.
How CloudNala can help
We help organisations make an AI assistant operable: trace design that respects data boundaries, evaluation suites that gate releases, a fully loaded cost-per-outcome measure, an operating rhythm with named owners, and clear conditions for switching capabilities off. The goal is that, every month, you can prove what the assistant is doing and whether it should keep doing it.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Production Readiness Review or write to us at consult@cloudnala.co.za