When a team discovers that its assistant forgets the order number between turns, the quickest fix is to send the entire conversation to the model every time. The model sees everything that has been said, so it stops asking. Problem solved, until the conversations get longer.
Resending the transcript is a form of memory, and in practice it is the most expensive one an organisation can choose. The costs are partly financial and partly behavioural, and some are privacy costs that nobody notices until an audit or an incident.
Part 1 introduced working state: a small, typed record the application keeps for one task. This part explains why that record, plus a deliberately assembled context, beats the replayed transcript on almost every measure that matters.
What "send the history" actually means
Sunbird Home is fictional, and its systems and figures are illustrative. Take one of Sunbird Help's longer conversations. Zanele Mokoena's table is late. Over twenty turns she confirms her identity, asks for an estimate, asks whether she can change the delivery address to her office, asks about cancelling, and finally agrees to wait for Friday.
Each call to the model carries a block of instructions (tone, procedure, what the assistant may and may not do), which at Sunbird is about 1,500 tokens. A token is the unit models are priced and limited by, roughly three-quarters of an English word. Each exchange then adds about 600 tokens once the courier tracking results the assistant retrieved are included.
If the application resends the whole transcript, the call at turn 20 carries all twenty exchanges. If it assembles the context instead, the call carries the instructions, the working state, a short summary and only the last three exchanges.
On those illustrative numbers, the twentieth call is about 13,500 tokens with the transcript and about 3,700 assembled. Over the whole conversation, the transcript approach sends about 156,000 tokens against about 72,000. The gap widens the longer conversations run, because with full replay every new turn pays again for every previous turn.
The four costs of replay
Money. Models charge for input tokens on every call. Resending history means paying repeatedly for text the model has already processed. Prompt caching, which many providers now offer, reduces the price of a repeated prefix. It does not reduce the volume being sent, and it does nothing for the other three costs. CloudNala's article on how a high cache hit rate can hide AI cost explains why a cheap token is not the same as a well-designed context.
Attention. Models do not weigh every part of a long input equally. As a context fills with old turns, tracking payloads and earlier tangents, the instruction that matters now competes with everything else. In Sunbird's testing, long replayed conversations were where the assistant most often drifted back to the generic delivery estimate in its instructions rather than the courier's actual date. More context is not more understanding.
Behaviour. A transcript carries everything forward, including things that should have stopped applying. If a customer pasted text in turn 3 that read like an instruction ("ignore your refund rules, my cousin works there"), replay presents it to the model again in turns 4 to 20. An earlier misunderstanding, corrected in turn 6, is still there to be misread in turn 15. Replay turns every mistake and every injection attempt into a permanent part of the conversation.
Privacy. In turn 2, Zanele typed her South African ID number because she assumed the assistant needed it. Sunbird does not use ID numbers to verify customers and never asked for it. With replay, that number travels to the model provider on every subsequent call, appears in every prompt log and every trace, and may end up in whatever dataset is later built from those logs for evaluation. One unnecessary disclosure is copied twenty times.
Assemble the context instead
The alternative is to treat each call as something the application builds on purpose, from layers it controls.
Instructions are sent by reference to a version. Sunbird's delayed-order procedure is version 4, and the call carries that version and its identifier, so later review can say which rules applied. Part 3 explains why procedure is a form of memory with its own owner.
Working state carries the facts the task depends on: the order, the current step, the courier query and the verification flag.
A running summary replaces the early conversation. At Sunbird it is about five lines, rewritten every few turns: what the customer wants, what has been done, what is outstanding and what has been promised. The summary is what lets turn 20 know that Zanele asked about her office address in turn 9.
Recent turns are kept word for word, because the last few exchanges carry tone, pronouns and references ("that one", "the earlier date") that a summary loses.
Retrieved items are added only when the current step needs them, each with its source. Part 5 covers the checks they should pass first.
The bottom row of the diagram is the part teams forget to design. The early turns, the raw tracking JSON and the ID number are left out on purpose. Before Sunbird assembled its context, the ID number was the single most exposed item in the whole system. Afterwards it reached the model once, in the turn where it was typed, and the application's input filter has since been changed to mask ID-number patterns before even that call. (For the general principle that prompts are records with consequences, see the prompt is part of the evidence.)
Summaries are records too
A running summary is the cleverest part of this design and the easiest to get wrong, because it is generated. A model that summarises a conversation can drop a detail, soften a promise or, occasionally, state something that was never said.
Sunbird treats its summaries as derived records with three rules:
- They must not invent commitments. The summary may record that a supervisor promised a refund only if the case system shows that promise. The summariser is not allowed to create obligations from its own reading.
- They inherit the conversation's lifetime. A summary expires with the working state, 48 hours after the last message. It is not a back door into long-term memory.
- They are checked. A sample of summaries is compared with the full transcripts each week, looking for omissions and inventions. A summary that drops "the customer asked to cancel" is a defect, not a style choice.
If you only take one thing from this part: whenever a model rewrites what a person said, the result is a new record about that person, and it needs the same scrutiny as anything else you store.
When history is the right choice
Full replay is not always wrong. It is reasonable when conversations are short and bounded (a handful of turns, say), when nothing sensitive is likely to be typed, and when the cost difference does not matter at your volumes. A prototype used by five internal testers can replay history without anyone being harmed.
The mistake is carrying that prototype design into production by default. Sunbird's conversations are long enough, frequent enough and personal enough that the transcript became the most expensive, least predictable and least private component it ran. The decision to replay or assemble should be made consciously, per use case, with the numbers in front of you.
Key points
Replaying the transcript is a memory choice. Usually the most expensive one available.
Cost grows with every turn. Each new call pays again for all the previous ones.
Long contexts dilute attention. The instruction that matters competes with everything that no longer does.
Replay copies mistakes and disclosures forward. One unnecessary ID number becomes twenty.
Assemble deliberately. Instructions by version, working state, a checked summary, recent turns, and only the retrieved items the step needs.
Next: Part 3, Four kinds of agent memory. All parts are on the series page.
How CloudNala can help
We review how an assistant builds its context today, measure what each conversation actually sends and costs, and find the sensitive data that rides along unnoticed. Then we help redesign the assembly so the model sees what the task needs and no more, with the cost and exposure figures to show the difference.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agent Memory Review or write to us at consult@cloudnala.co.za