Stop sending every agent task to the most expensive model

A resident types "what time does the customer care office open on Saturday?" into a municipal help service. A capable small model can answer that from an approved page for a fraction of what a frontier model costs. Ten minutes later the same resident writes "it still says R0". Read on its own, that message looks like the easiest question of the day. Read with the session in view, it is a follow-up to a failed balance lookup, on a rates account, after an integration error. It is the hardest turn in the conversation.

That is the problem with most model-routing advice. It treats each request as an isolated prompt and asks which is the cheapest model that looks capable of answering it. An agent is not a stream of isolated prompts. It carries history, tool results, unresolved errors and risk, and switching models has a cost of its own. Routing for an agent is a control decision made repeatedly over a session, not a price comparison made once per message.

This article explains what changes when you route a stateful agent, where the hard rules belong, what the router needs to know, and how to tell whether routing saved anything. It uses a small public-safe lab we built for this series, run on synthetic sessions with simulated models for Gamtoos Assist, the help service of Gamtoos Valley Municipality, a fictional Eastern Cape municipality.

Treat the router like a triage desk, not a price list

A supermarket price comparison looks at one item and picks the cheapest shelf. A hospital triage desk looks at the patient: what happened before, which tests already failed, how serious the risk is. A routine follow-up can go to the general queue. A complicated case with prior results and a failed treatment keeps its history and may need a specialist, even if the patient's latest sentence sounds simple.

A model router for an agent should behave like the triage desk. The question is not "which model can answer this sentence?" but "which model should own this turn, given what this session has already been through?"

What the research says, and what it does not

We found the clearest recent treatment of this through the AI Engineering newsletter's summary of context-guided routing (Sumanth P, 30 October 2026). The underlying paper, Whose turn is it? Context-guided model routing for stateful agents (Parashar and colleagues, a team including Microsoft researchers; version 1, 13 October 2026), studies zero-shot, cost-aware routing at the user-turn boundary for agents that keep state.

Its findings, as reported in the paper:

Giving the router access to session history improved task pass rate at the same offloading rate for every model pair tested, and kept history-aware routing within about one point of always using the expensive model.
The cheaper model's per-task cost advantage narrowed from 6.8 times to 3.2 times on longer sessions.
Switching from an expensive model to a cheap one recovered its prompt-cache cost within two model calls. Switching upwards did not break even under the conditions evaluated.
The authors conclude that routing for agents is a sequential decision problem.

Two cautions matter before anyone builds a business case on those numbers. The benchmarks are multi-turn spreadsheet tasks (SheetCopilot and SpreadsheetBench 2) with persistent workbooks. A citizen help service, a claims workflow or a coding agent may behave differently. And, as the paper itself notes, the two benchmarks also differ in workbook size and domain, so the 6.8 times and 3.2 times comparison is descriptive rather than a pure measure of session length. Treat the paper as evidence that history and cache effects matter, not as a forecast of your savings.

Hard gates come before the router

The first design rule is that some decisions are not routing decisions at all. Before any probabilistic or model-based router runs, deterministic gates should remove the options that are never acceptable for the turn:

Data residency and approved inventory. A model or region the organisation has not approved for this data class is not a candidate, however cheap.
Tool and output support. If the turn needs a tool call or a structured output the model does not support reliably, it is not a candidate.
Consequence class. A turn that could change a citizen's record, decline an application or move money can only reach a model approved for that consequence class.
Human decision points. Some outcomes are not the model's to give. A likely adverse decision on an indigent-subsidy application goes to an official, whichever model drafts the explanation.

Hard policy gates run first and are not probabilistic: a high-consequence turn can only reach a model approved for it, and a likely adverse decision goes to an official whatever the model says. Only then does the router choose, using session state as well as the latest message. Escalation to a person happens after the model, with the evidence attached.

These gates are policy, written as code, tested like code and owned by someone with authority over the policy. They should never live inside a prompt to the router. In the lab, the always-small policy was overridden on 162 of its 826 turns (about 20%) across 200 sessions because high-consequence turns were forced onto the approved model. That count is itself useful: it shows how much of the workload a "cheap model only" plan could never have handled.

Route at a boundary you can explain

The paper's main router assigns a model once per user turn: the chosen model then runs the whole loop of model calls and tool calls for that turn. That is a good default for three reasons.

It keeps behaviour explainable. "This turn ran on the large model because the previous turn failed" is a sentence a support analyst can read in a trace. A model swap halfway through a tool sequence is much harder to reason about.
It respects the cache. Every switch can force the accumulated context to be processed again at full price. Switching less often, at deliberate points, keeps that cost visible.
It matches how work is accepted. People judge whether a turn was handled, not whether a single model call was. Routing at the turn lets you measure cost and outcome on the same unit.

Workflow stages are another good boundary. Intake can run on a small model, investigation on a larger one and the final drafted explanation on whatever passes its evaluation. The point is to pick the boundary deliberately, not to let the router flip models on every call.

Give the router the minimum session state

A session-aware router does not need the whole transcript. It needs a handful of facts about what has already happened.

A follow-up such as 'it still says R0' looks trivial in isolation. With the session in view it is a recovery from a failed lookup, which is the hardest kind of turn. The router does not need the whole transcript, only a few facts about what has already happened. The lab's session-aware policy uses the last failure, open risk flags and the cache holder; the other two are recommended for real systems.

In the Gamtoos Assist lab, the session-aware policy reads a subset of them: whether the last turn failed, whether a risk flag is open and which model holds the cache. Tracking unresolved tool errors and the workflow stage is our recommendation for real systems; the lab does not model them. It escalates when the last turn failed or a risk flag is open. Once it has escalated, it steps back down only after two clean, easy turns, because a downward switch recovers its cache cost quickly while an upward one is a quality decision, not a saving. Everything else falls back to the latest-message rule. That is deliberately simple. A router you cannot explain is a router you cannot govern.

Keep the session state small and typed: a boolean for the last failure, a list of open tool errors, a set of risk flags, the workflow stage and the model that holds the cache. Do not pass free-text summaries of the conversation to the router and hope it infers the same things.

Measure cost per accepted outcome, not price per token

The only fair test of a routing policy is to run it against the alternatives on the same work and compare what was accepted and what it cost. We did that in the lab, with four policies on the same 200 synthetic Gamtoos Assist sessions.

Simulated output from the lab, not model performance. Routing on the latest message put most turns on the cheap model and saved almost nothing per accepted session: recovery turns failed, sessions were abandoned and cache was lost on every switch, so total spend fell by about 40% but accepted sessions fell by almost as much. Session-aware routing kept the pass rate within a point of always-large at a modest saving. Costs are illustrative credits.

These are simulated results. The lab uses a deterministic simulator with assumed success rates and illustrative prices, with the large model priced at ten times the small one. The numbers show how to measure. They say nothing about how any real model performs. With that caveat, the pattern is instructive:

Always small looks cheapest per accepted session, but it accepted fewer than half the sessions. And it could only run at all because the hard gates sent the high-consequence turns to the large model.
Latest message only put about 60% of turns on the small model and still cost almost as much per accepted session as always using the large one. Follow-ups that looked easy were recovery turns, the small model failed them, sessions were abandoned after repeated failures, and the cache was lost on every switch. Total spend fell by about 40%, but accepted sessions fell by almost as much.
Session-aware stayed within one point of always-large on pass rate, at about 6% lower cost per accepted session.

A 6% saving will not transform a budget, and in the real world the figure could be larger, smaller or negative. The useful lesson is that the naive router saved almost nothing per accepted session while accepting only 59% of sessions against 97.5%. Without a comparison against fixed baselines, the team would have celebrated a 60% offload rate and never noticed. This is the same discipline CloudNala argued for in cost per accepted outcome, not per prompt: the unit of value is the outcome someone accepted, not the token.

Log every routing decision

If a router chooses models, every choice is a production decision and needs a record. For each turn, log at least:

the model selected and the policy version that selected it;
the reason, as a typed value ("last turn failed", "risk flag open", "gate: consequence class");
whether the prompt cache was reused or lost;
latency and cost for the turn;
whether the turn's outcome was accepted, corrected or escalated.

That record answers the questions a finance team, an auditor or an incident reviewer will ask later. It also makes the router tunable: you can see which reasons lead to failures and which escalations were wasted. Logging does not need a new platform. If the organisation already runs a model gateway, the routing reason belongs alongside the gateway's own records. The model gateway is not the AI platform explains where those records should live and who owns them.

Plan the fallbacks before the first outage

Routing changes the failure modes. Add explicit fallbacks for:

Model unavailability. Decide in advance whether a high-consequence turn waits, fails safe to a person or uses a second approved model. Do not let a fallback quietly route it to a model that failed the gate. Who owns the retry? covers why retries need one owner.
Invalid structured output. One validation failure should trigger a defined retry, then escalation, not an open-ended loop between models.
Repeated tool failure. After a set number of failures the turn stops and goes to a person, whichever model is running.
Risk discovered mid-turn. If a tool result reveals a conflicting identity record, the risk flag is raised for the rest of the session, so the next turn is routed with it.

Running the lab

The lab is small and has no dependencies. It needs Node.js 22 or later and makes no network calls. npm test runs 14 deterministic tests, and npm start runs the routing comparison along with the two other experiments in this series, writing results to a results folder. Every Gamtoos Valley Municipality session, citizen and figure in it is fictional. To measure real models, replace the simulator with an adapter for your provider, keep the same interface, and run your own sessions with a labelled golden set and repeated runs. The lab is open source under the MIT licence: practical-agent-engineering-lab on GitHub.

Key points

Routing for an agent is a sequential decision about a session, not a per-prompt price lookup.
Hard gates for residency, tool support, consequence class and human decisions run before any router, and they are owned as policy.
Route at an explainable boundary, such as a user turn or workflow stage, and give the router a small, typed set of session facts.
Compare every routing policy against always-strong and always-cheap baselines on the same work, measured by cost per accepted outcome.
Published routing results come from specific benchmarks. Use them as evidence that history and cache matter, not as a savings forecast.

Next: Part 2, You do not control the agent until you control the loop. All parts are on the series page. For the evaluation side of choosing a model, see use the smallest model that reliably passes the evaluation and prompt caching is an architecture decision.

How CloudNala can help

CloudNala reviews agent and assistant designs before the routing logic hardens into production. We map which turns carry real consequence, write the hard gates as testable policy, define the session facts a router may use, and set up a comparison against fixed-model baselines on your own representative work. You leave with a routing decision record, a measurement plan based on cost per accepted outcome, and a clear view of where a cheaper model is genuinely safe to use.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agent Engineering Review or write to us at consult@cloudnala.co.za