The finding that mattered most in Kudu's first red-team exercise did not involve any clever prompt. A tester, logged in as a patient with a linked dependant, asked the assistant to move "my prescription" to a different store, and gave the request number of a prescription belonging to a different patient altogether. The assistant called the change-store tool with that number. The tool checked that the number existed and that the session belonged to a patient with dependants, and made the change.
The model had done exactly what it was asked. The weakness was in the tool's authorisation check, and no amount of testing what the assistant said would have found it. It showed up only because the exercise measured what the system did.
Kudu Pharmacies is fictional, and its systems and figures are illustrative. As described in Part 10, Kudu's in-app assistant answers questions about repeat prescriptions and can start two workflows: requesting a repeat and changing the collection store. Patients can manage prescriptions for linked dependants. Every exercise in this part ran in a test environment with synthetic patients, with the synthetic-data rules from Part 9.
What changes when the system can act
Most red-teaming of language models has focused on outputs: can the model be made to say something harmful, leak its instructions or produce prohibited content. That still matters. A tool-using agent adds a second question that matters more: can anyone make the system change something it should not?
The OWASP Top 10 for LLM Applications (2025 edition) names both sides. Prompt injection is LLM01, and excessive agency, where a system has more functionality, permission or autonomy than it needs, is LLM06 (checked on 5 November 2026). OWASP has since published a separate Top 10 for Agentic Applications (announced 9 December 2025), whose categories include ASI02, Tool Misuse and Exploitation, and ASI03, Identity and Privilege Abuse. The finding that opens this article falls squarely in the second. Research has shown how the two combine. Greshake and colleagues described indirect prompt injection in 2023: instructions planted in content an application retrieves can steer the model without the attacker ever talking to it. The AgentDojo benchmark (Debenedetti and colleagues, 2024) built 97 realistic agent tasks and 629 security test cases to measure how often injected instructions in tool results succeed.
CloudNala has already published the design review side. Agents change the risk boundary lists the questions to ask of an agent's action path, and The MCP security review sets out nine questions before a tool server goes live. This part is about the testing practice that produces evidence for a release: how to derive attack cases, run them, judge them and keep them.
Start from the tools, not from a list of jailbreaks
Generic jailbreak collections are a reasonable smoke test and a poor red-team plan. They test the model's general resistance, not your system's specific exposure. The better starting point is the tool inventory.
For each tool the agent can call, ask what an attacker would want it to do that it should not. Kudu's two workflow tools and one read-only tool produced a short list:
| Tool | Abuse an attacker would want | Harm if it succeeds |
|---|---|---|
| Request repeat | Request a repeat for another patient; request one that is not due; request one for a medicine that needs pharmacist review without it being held | Medicine released to the wrong person or too early |
| Change collection store | Move another patient's prescription; move a held prescription to bypass review; move many prescriptions at once | Medicine collected by the wrong person; disrupted stock |
| Read request status (read-only) | Read another patient's status or medicine names | Disclosure of health information |
Each row becomes a set of abuse cases. Each abuse case is then attempted through every channel an attacker could use.
Attack every input channel
An agent takes instructions from more places than the chat box. Kudu's red-team plan covers four channels.
- The user turn. Direct requests, role claims ("I am the pharmacist on duty"), urgency, and requests that split a forbidden action across several turns.
- Retrieved content. Kudu's assistant retrieves store pages to answer questions about hours and stock. Store pages are edited by store staff through a content system. A planted instruction in a store page is an indirect injection, and the test plants one.
- Tool results. A result from one tool can carry text that the model treats as an instruction for its next step. Test cases plant instructions in fields the model reads, such as a free-text prescription note.
- Other sessions. Cases where one patient's action could affect another patient's data, including through linked dependants, which is where Kudu's real finding came from.
Each channel needs its own cases because defences differ. A model may resist a direct instruction to bypass a hold and still follow the same instruction when it arrives inside a store page.
Judge success by state, not by words
The single most important design choice in a red-team suite for an agent is the oracle: how you decide whether an attack worked.
Judging by the agent's reply is unreliable. An agent can refuse politely while a tool call it made earlier has already changed a record. It can also describe an action it never took. Kudu's suite therefore judges every case by checking state in the test environment after the run:
- Did any prescription belonging to a different synthetic patient change?
- Did any held request move, release or lose its hold?
- Did any response, log or trace contain another patient's medicine names?
- Did any tool receive a call outside its contract?
These checks are deterministic queries against the test database and logs. They do not need a model to grade them, and they cannot be talked out of a result. AgentDojo takes a similar approach, checking the environment's state to decide whether an attack's goal was achieved, alongside whether the user's legitimate task still succeeded. That second measure matters too: a defence that blocks every attack by refusing every request has not made the system safe, only useless.
Who does the red-teaming
The team that built the assistant should not be the only team that attacks it. Builders test the attacks they designed against. Kudu uses three sources of cases:
- The product team writes abuse cases from the tool inventory as part of building each tool. This is the baseline and runs on every change.
- The security team runs a structured exercise before the first release of any new tool and quarterly thereafter, adding cases from current public research and their own threat model.
- Pharmacists contribute cases from the way real patients behave, including the ways people genuinely try to get medicine sooner. Many of the most useful cases look nothing like a hacker's attack.
Findings from any source are triaged the same way.
Fix the control, then keep the case
When a case succeeds, the reflex is to add an instruction to the prompt: "never change a prescription that is not the user's own". That instruction may help. It is not a control. A prompt is advice to a model, and the next phrasing of the attack may get around it.
Kudu's triage rule is that every successful attack is fixed at the lowest layer that can enforce it, and the case then joins the permanent suite.
- The dependant finding was fixed in the tool: the change-store tool now checks that the request belongs to the session patient or to one of that patient's linked dependants, using the patient registry, not the model's claim. This is the capability and authority distinction applied in code.
- The planted store-page instruction never succeeded, because no store-page content can reach a tool call without a confirmation step. The case stays in the suite to keep it that way.
- A multi-turn case that persuaded the assistant to describe a held request as "almost ready" was fixed by generating status wording from the request state through a template, not by the model, for held requests.
Every case is versioned with the suite. Under the routing from Part 10, adversarial cases run in full whenever a tool, a tool's permissions, the retrieval sources or the model changes, and a sample runs on other changes. Any successful attack on a high-harm case blocks the release.
What to report in the release record
Red-team results belong in the release record in the same form as other evidence: what ran, against which version, with what result.
- Number of cases by tool and channel, and how many succeeded.
- For each success: the case, the harm level, the fix, the layer where it was fixed and the re-test result.
- Task success under attack: the share of legitimate tasks that still completed while attacks were present.
- Cases added since the last release and their sources.
At Kudu, the first full report showed 212 cases, three successes, all fixed at the tool or template layer, and legitimate task success unchanged. That is a short paragraph a release manager can rely on, and a much stronger statement than "security had a look".
Key points
- For a tool-using agent the critical red-team question is whether anyone can make the system change something it should not, not only what it says.
- Derive abuse cases from the tool inventory, then attempt each through every channel: user turns, retrieved content, tool results and other sessions.
- Judge attacks by checking state in the test environment, and measure whether legitimate tasks still succeed.
- Fix each successful attack at the lowest layer that can enforce it, usually the tool or harness, not the prompt.
- Keep every case in a versioned suite that runs whenever tools, permissions, sources or the model change, and report results in the release record.
Next: Part 12, Every incident becomes a test case. All parts are on the series page. For the controls that limit damage when a test misses something, see Preventing expensive and destructive tool calls.
How CloudNala can help
CloudNala builds red-team suites for tool-using agents. We derive abuse cases from your tool inventory, design state-based oracles for your test environment, run a structured exercise with your security team, and set up the triage and release reporting that turn findings into fixed controls and permanent cases, so that each release carries evidence about what the agent can be made to do.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Release Evidence Review or write to us at consult@cloudnala.co.za