A 98.5 percent cache hit rate reads as a job well done. Almost everything being sent is being recognised as repeated and discounted accordingly. By the usual standards of an efficiency metric, that is close to the ceiling.
It was also, in the workload Andrew Baker instrumented, sitting on top of a serious cost problem. Both things were true at once, and the reason they can be true at once is that the metric answers a different question from the one people think it answers.
A discount on waste is still waste
If you get ninety percent off something you never needed to buy, you have not saved money. You have spent ten percent of an amount that should have been zero.
Cache hit rate tells you what proportion of the repeated context received a discount. It does not tell you whether that context needed to be there. Those are separate questions with separate levers, and only one of them makes the volume smaller.
This is why the metric is comforting in exactly the situations where it should not be. As a session grows and the same enormous context is resent turn after turn, the hit rate goes up, because the proportion of the payload that is repeated increases. The metric improves as the problem worsens. That is not a subtle statistical trap, it is a direct inversion, and it explains how a team can watch a dashboard for months and see nothing wrong.
Four numbers from one investigation
Read as a set rather than individually, these tell a coherent story that no single one of them tells alone.
Cache hit rate of 98.5 percent. Almost all repeated content is being discounted. Good, and irrelevant to the size of the payload.
Input-to-output ratio of 440.8 to 1. For every token of output produced, roughly 441 tokens of input were sent. This is the number that should have raised the alarm, because it describes how much material is being carried per unit of actual work.
Context delta per turn. Not a headline number but the diagnostic one, because it shows the exact turn at which context stepped up and lets you go and read what happened there.
Top ten percent of sessions, 63.2 percent of spend. The distribution. Two thirds of the money is in a small number of sessions, so the fix is targeted rather than cultural.
Which metrics to watch, and what each is for
A short field guide, since these terms get used loosely.
Cache hit rate. Watch it like an error rate: a sudden drop means something changed in how context is being constructed and is worth investigating. A high value means nothing on its own. Never report it without a volume figure beside it.
Input-to-output ratio. The best single indicator of context discipline. It is intuitive to non-technical stakeholders, which makes it useful in a governance forum, and it is hard to game.
Context delta. The per-turn change in context size. This is the diagnostic instrument: it converts "the session was expensive" into "the session became expensive at turn 43, when the agent read the whole schema directory".
Cost per turn. Useful for spotting the point where a session's economics changed, and for setting an alarm.
Session total. The unit people intuitively understand, and the one to use when asking someone to change behaviour.
Top expensive sessions. The action list. Not a summary of them, the actual sessions, read by a person.
Two different levers
The practical conclusion is worth stating plainly, because it resolves an argument that comes up in most teams.
Caching is a price lever. It reduces what you pay for context you have decided to carry. It is worth having, it is largely automatic, and it requires no behaviour change.
Session isolation is a volume lever. It reduces how much context exists to be carried at all, by ending sessions deliberately, isolating exploratory work, and starting new tasks clean. It requires a small behaviour change and it usually produces a much larger effect.
They are not alternatives, and a team that has done the first is not partway through the second. We cover the volume lever in session hygiene and subagents as cost isolation.
What good looks like
Cache hit rate never reported alone, always with total volume beside it.
Input-to-output ratio tracked as the headline context-discipline number.
Context delta available per turn, so a spike has an address.
Reporting that shows the distribution rather than the average.
An explicit understanding, shared with whoever reads the numbers, that a high hit rate is not evidence of efficiency.
Practical checklist
- Put a volume figure next to every cache hit rate you report
- Track input-to-output ratio and set a threshold that triggers a look
- Capture context delta per turn, not just session totals
- Report the top ten sessions, not the mean
- Treat a rising hit rate on a growing session as a warning, not reassurance
- Separate price levers from volume levers when discussing fixes
How CloudNala can help
Most of the value here comes from changing what gets reported rather than changing what gets built. We help teams put the right pair of numbers in front of the right audience, so that a cost conversation starts from context volume and distribution rather than from a model price card. That usually reframes the problem within a single meeting.
The figures in this article come from Andrew Baker's August 2026 write-up on instrumenting coding agents, which is worth reading in full.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za