Prompt caching is usually introduced as a setting: switch it on and the bill goes down. Sometimes it does. Sometimes it goes up slightly, and nobody notices because the cache dashboard shows a healthy hit rate somewhere else in the estate.
Caching is better understood as an architecture decision. Whether it saves money depends on how prompts are assembled, how often the same content is reused within a time window, and how the application is split across tenants and workspaces. None of those are provider settings. They are choices the team makes, usually without realising they are making them.
This part explains the mechanics, with every provider behaviour checked against first-party documentation on 27 October 2026. Providers change these details often, so treat the specifics as a dated snapshot and check again before relying on them. The earlier article Why a 98% cache hit rate can still hide an expensive AI workflow makes the complementary point: caching reduces the price of context you resend, not the volume. Both are true at once.
Tugela Claims Services is fictional, and its systems and figures are illustrative. Tugela's triage agent, from earlier parts of this series, sends each claim to a model with tool definitions, a system prompt containing triage rules and a claims-handling guide, the policy wording of whichever of its seven insurer clients the claim belongs to, and then the claim itself.
What a cache actually stores
When a model processes a prompt, it does a large amount of computation on the input before it writes anything. If the start of the next prompt is identical, that work can be reused instead of repeated. That is all a prompt cache is: stored computation for an exact prefix.
Three properties follow, and they drive every design decision below.
It is a prefix match. The cache can reuse the beginning of a prompt up to the first point where anything differs. A one-character change near the top invalidates everything after it.
It expires. Stored computation is held for a limited time. One major provider's default on 27 October 2026 was five minutes, refreshed at no extra cost every time the entry is used, with a one-hour option at a higher write price. Another's newest models defaulted to a minimum lifetime of 30 minutes, with a 12-hour option.
It has a minimum size. Prompts shorter than a threshold are not cached at all. Published minimums ranged from 512 to 1,024 tokens on current models, and higher on some older ones. No error is returned; the request simply runs uncached.
According to both providers' documentation, the cache stores the result of computation, not a copy that can be returned to another customer. One states that caches hold key-value tensors rather than prompt text. Both state that caches are not shared between organisations.
The pricing, and why it is not automatically a saving
Caching changes the price of input tokens in two directions.
Writing to the cache costs more than normal input. One provider charged 1.25 times the normal input price for a five-minute cache write and 2 times for a one-hour write. The other charged 1.25 times for its 30-minute writes and 2.5 times for 12-hour writes on its newest models, with no write premium on earlier ones.
Reading from the cache costs much less. Both published a read price of one tenth of normal input on most models, and one twentieth on some of their current flagship models.
So a cached prefix costs a little extra the first time and much less each time after that, but only if there is an "after" within the expiry window.
The arithmetic for a five-minute cache at these rates:
- Written and never reused: 1.25 times the normal cost. Caching made it 25% more expensive.
- Written and read once: 1.35 times, against 2 times uncached. A 32% saving.
- Written and read three times: 1.55 times, against 4 times uncached. A 61% saving.
For a one-hour write at 2 times, a single reuse does not break even (2.1 against 2.0). It needs two.
The design question is therefore not "should we cache" but "which content is reused within the window, and how often?"
Prompt order is the main lever
Because caching is a prefix match, the order in which a prompt is assembled decides what can be reused. The rule is to put the most stable content first and the most volatile last.
One provider documents its cache hierarchy explicitly: tool definitions first, then the system prompt, then messages. A change at any level invalidates that level and every level after it. Changing a tool definition therefore invalidates everything, while adding an image to a message invalidates only the messages.
Tugela's first implementation broke this rule in a way that is extremely common. The system prompt began with a line stating the current date and time and the claim reference number, because a developer thought the agent should "know what it was working on". Every call therefore had a unique prefix from the second line onwards. The cache dashboard reported writes on every call and almost no reads. Caching was switched on, and it was costing 25% extra on the cached portion.
The fix took an afternoon. The date and claim reference moved into the claim message at the end. Tool definitions and the triage rules became a stable prefix shared by every call. The insurer's policy wording became the next layer, shared by all calls for the same insurer.
Timing: where reuse actually happens
The second lever is how close together the reuse happens. This is where agent design and caching interact in a way that is easy to miss.
Within one agent run, reuse is almost guaranteed. Tugela's nine calls per claim happen within a minute or two, and every call after the first carries the same instructions, tools, policy wording and claim. This is the best caching case there is: many reads per write, all well inside any expiry window. For multi-step agents, most of the caching benefit comes from here.
Across runs, it depends on volume. Tugela processes about 18,000 claims a month, roughly one a minute during working hours. The shared prefix of tools and triage rules stays warm all day. The insurer layer is different: the largest client sends hundreds of claims a day and stays warm, but the smallest sends about 300 a month, often with gaps longer than five minutes. For that client the insurer layer is frequently written and expires unread.
Across the night, nothing survives. The first call each morning pays the write price again. That is fine; it is one write against thousands of reads.
The practical design for Tugela was to cache the shared prefix and the insurer layer for large clients on the default window, and to place an explicit cache boundary before the insurer layer for small clients so they did not pay write premiums for content that would expire unused. Whether a longer window would pay for a small client is simple arithmetic: at a 2 times write price it needs at least two reads within the hour.
Tenancy and privacy
Caching raises a reasonable question for any business that processes data for several clients in one system: could one client's content leak into another's answers?
The cache does not return stored text in response to a different prompt. It reuses computation only when the prefix is identical. Providers isolate caches between organisations, and one documents isolation between workspaces within an organisation on its own platform, though only organisation-level isolation on some partner clouds. One also states that caches cannot be reused across regional processing boundaries, which matters if you have pinned inference to a region.
The real tenancy risk sits in prompt assembly, not in the cache. If a bug places one insurer's policy wording into another insurer's claim prompt, that is a data-handling failure whether or not caching is enabled. The same discipline that keeps tenants separate in retrieval, covered in Permission-aware retrieval, applies here.
Where a contract or regulator requires processing for different clients to be kept apart, separate workspaces or accounts per client make the separation explicit, at the cost of less cache sharing. That is a legitimate trade-off to make deliberately.
A caching design checklist
- Order the prompt by stability. Tools and fixed instructions first, then slowly changing reference material, then the case.
- Remove volatile values from the prefix. Timestamps, request identifiers, user names and session counters belong at the end.
- Check the minimum size. A short system prompt below the threshold will never cache, silently.
- Measure reads per write, by layer. A layer with fewer than one read per write is costing money.
- Choose the window from the traffic pattern. Longer windows cost more to write and need more reuse to pay back.
- Treat a tool-definition change as a cache reset. Release tool changes deliberately, not mid-morning on a busy day.
- Separate tenants where the contract requires it, and accept the lower hit rate as the cost of that requirement.
- Re-check provider documentation before relying on any of the numbers above. These details change often and without much notice.
Next: Part 5, Batch, realtime and the rand. All parts are on the series page.
How CloudNala can help
We review how AI applications assemble their prompts, where volatile content is breaking reuse, and which layers are written more often than they are read. The result is a prompt layout and caching configuration that matches real traffic, with measurement by layer so the saving can be verified rather than assumed.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an AI Cost Review or write to us at consult@cloudnala.co.za