When the tool fails but the answer looks complete

Suppose an agent is asked to research criticism of a product on two social platforms. The search on the first platform succeeds. The search on the second fails. The agent still writes a fluent report.

There are two possible conclusions about the second platform. Either no critical posts were found there, or the platform could not be searched and the evidence is incomplete. Only the second is supported.

Written down, the distinction is obvious. It is easy to lose inside an autonomous loop. The model sees several successful results, writes a coherent narrative and treats the missing source as if it contained nothing. The finished prose can sound more certain than the work behind it deserves. Partial failure is normal in any distributed system; what agents add is a component that is very good at writing around the gap.

The brief that nearly cut a promotion

Weaverbird Agri is fictional, and its systems and figures are illustrative.

In Part 2 we looked at the trace of one Scout run. Scout is the market-research agent at Weaverbird Agri, a Bloemfontein-based agricultural inputs distributor. In that run, the social media search was rate-limited by its provider, retried, and failed. The price search, memory retrieval and synthesis all worked. The citation check passed, because every citation in the brief pointed to a document Scout had actually retrieved.

The brief's competitor section said: "No competitor promotions found on social media this week."

A regional sales manager read that sentence as good news. Weaverbird had an early-season maize seed promotion planned for its Northern Cape depots, and with no competing offers in the market, the manager proposed scaling it back and keeping the margin. Two days later a depot manager in Kimberley forwarded a competitor’s promotional flyer that a farmer had passed on. It had been on social media all week.

Nothing in the brief was technically false. Scout had found no promotions on social media, because it had not searched social media. The analyst who reviewed the brief had no way to tell, since the brief did not say. The trace knew, but nobody reads a trace before reading a brief.

Failure has several shapes

An explicit error is the easy case. Most partial failures are quieter, and each one invites a dangerous interpretation.

Scout's failure was a rate limit read as an exhaustive search. A permission failure read as "record does not exist" is the same mistake in a different system, and it is common: an agent that cannot see a customer's file is easily convinced the customer has no file.

Define the required evidence before the run

The fix starts before the agent runs. Each kind of task should state what evidence it needs, which sources are required, which are optional, and what happens when one fails. Weaverbird now gives Scout a research contract for each section of the brief. For the competitor section it reads:

research_contract:
  section: competitor_promotions
  required_sources:
    - social_search
    - competitor_websites
  optional_sources:
    - trade_press
  freshness_days: 7
  minimum_independent_sources: 2
  citation_required_for_material_claim: true
  on_required_source_failure: block_section
  on_optional_source_failure: disclose_limitation

Social media is a required source for this section because that is where competitors in this market actually run promotions. For the fertiliser price section it is optional at best. The contract records that judgement, so the agent does not have to make it.

With a contract in place, every run ends in one of five states, rather than "done":

• Successful and complete: the required coverage was achieved.
• Successful with limitations: the answer is useful, but one or more optional sources failed.
• Incomplete: required evidence is missing.
• Failed: no reliable result was produced.
• Escalated: a person's judgement is needed.

The failure-aware loop

The loop has one rule that agents tend to break: a failed call ends in one of only two places. Either the call is retried, because repeating it is safe, or the failure is recorded as a limitation that travels with the answer. It never just disappears.

Retries need boundaries. Retrying an idempotent search is safe, though it costs money, as Weaverbird's CFO found out in Part 1. Retrying a payment, a message, an account change or an infrastructure deployment may not be. Depot Desk, Weaverbird's dealer-service agent, drafts credit notes, and a retried draft is a duplicate credit note waiting to be approved. Consequential tools need idempotency keys, transaction references or a confirmation step before anything is repeated. Part 5 comes back to this when it looks at tool access.

Five states the prose tends to collapse

The deeper problem is that there are at least five different things "nothing was found" can mean, and each supports a different sentence.

The architecture has to keep these states apart all the way to the output:

• No evidence exists. Rarely provable, and almost never what an agent actually knows.
• No evidence was found in the sources that were searched.
• Evidence was not searched for.
• Evidence could not be accessed, because of a failure, a permission or a limit.
• Evidence was found but could not be verified.

Scout's brief reported the second state when the fourth was true. Making the distinction explicit costs almost nothing. The research contract tells the agent which sources were required, and the trace tells it which of them succeeded. The generation step only has to be told to use both.

Test the claim, not just the error handler

A weak test of this behaviour asks: was the failed call retried or mentioned? Scout would have passed it. The brief did mention social media.

A stronger test asks:

• Did the final answer name the failed source?
• Did it avoid turning a failure into a zero-result claim?
• Did its confidence fall appropriately?
• Were conclusions that depended on the missing source blocked?
• Were citations limited to evidence actually retrieved?
• Did the system avoid unsafe retries?
• Was the incident visible to the right owner?

That catches the subtle case. A report can say "no posts were found" and technically cover the step, while still presenting a tool failure as a research finding. Weaverbird added the Kimberley brief to Scout's test set exactly as it happened: a rate-limited social search, with a pass condition that the competitor section is withheld and the reason given. Part 6 covers how cases like this become a release gate.

Show the uncertainty to the reader

The person reading the answer should not need a trace viewer to find out it is incomplete. Scout's briefs now open with a coverage line:

Coverage: Price history, supplier price lists and competitor websites searched successfully. Social media search failed after two attempts because the provider rejected the request. The competitor promotions section is withheld this week; check with depot managers before changing planned promotions.

For high-impact work, the interface can also show:

• evidence coverage, as sources attempted against sources completed;
• the freshness date of the evidence;
• failed or excluded evidence;
• human review status;
• confidence with a plain-language reason.

Avoid a decorative confidence score with no calibration behind it. "82% confident" means nothing unless the score has a defined method and a demonstrated relationship to how often the agent is right. A plain sentence about what was and was not checked is more honest and more useful.

The operating lesson

Partial failure is normal in distributed systems. Agentic systems make it harder to see because the language model can write around the gap, and its writing is good enough that people believe it.

So the architecture has to preserve the difference between no evidence exists, no evidence was found, evidence was not searched, evidence could not be accessed and evidence was found but not verified, all the way from the tool call to the sentence the reader acts on. The trace keeps the first half of that chain. The research contract, the generation instructions, the evaluation and the interface keep the rest.

The short version

Fluent is not complete. A model will write around a failed source unless told not to.

Define required evidence up front. A research contract says which sources matter and what happens when one fails.

Retry or record, never drop. Every failed call is either safely retried or carried as a limitation.

Test the claim. Check whether the answer misrepresents a failure, not only whether the failure was mentioned.

Tell the reader. A coverage line beats a confidence score.

A complete-looking answer is not evidence of a complete process.

Previous: Part 2, a trace is not an evaluation. Next: Part 4, memory is a governed data product. See every part on the series page. For how agents use tools in the first place, see how AI agents actually do work.

How CloudNala can help

We help teams make partial failure visible: research contracts that define required evidence for each task, failure-aware loops with safe retry rules and idempotency for consequential tools, evaluation cases built from real incidents, and interfaces that tell the reader what was and was not checked. We usually start by replaying a handful of your agent's recent runs with one dependency switched off, which shows quickly whether failures are reaching the reader or disappearing into prose.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an Agent Evidence Review or write to us at consult@cloudnala.co.za