The meeting is familiar. A supplier, or an internal team, presents a platform. The slides say the platform contains "multiple intelligent agents" that "autonomously resolve" service requests. The demonstration is smooth. Someone on the client side asks whether the agents are secure, and the answer is "yes, enterprise-grade".
Then a different question is asked: "What can these agents observe, what can they decide, what can they change, and what can you prove about how they behaved last week?" The room changes. The conversation moves from marketing language to operational reality, and within ten minutes everyone knows much more about what is being offered.
This article is a reusable version of that second question. It is written for CIOs, CTOs, business executives, architects, procurement teams, risk and compliance leaders, product owners and internal audit. It closes the Agent Claims Decoded series, but it is designed to stand on its own: you can take it into a supplier meeting, an investment committee or an internal design review.
The firm position behind it is simple. Do not reject the word agent. Make it testable.
How to use the question bank
The questions are grouped into twelve domains. For each domain, ask the questions, then ask for the evidence listed. Rate the evidence you are shown, not the confidence of the answer.
| Rating | Meaning |
|---|---|
| 0, claim only | The answer is verbal, vague or based on a demonstration |
| 1, designed | Architecture, policy or intended behaviour is documented |
| 2, tested | Evidence exists from controlled evaluation, including failure cases |
| 3, operated | Production evidence, ownership, monitoring and incident learning exist |
Score each domain separately and do not average them. A polished user experience cannot compensate for missing authority, security, evaluation or audit evidence. A single 0 in one of those domains is reason enough to pause.
For an early-stage pilot, ratings of 1 or 2 are normal and acceptable. What matters is that the team knows which rating applies and has a plan to move it. For a production purchase that will touch citizens, customers or money, the critical domains should be at 2 or above before contract and at 3 within an agreed period.
1. Purpose and outcome
- What exact business outcome does the agent own?
- Who initiates its work and who receives the result?
- What is the start condition, and what counts as completion?
- Is it answering, recommending, preparing, executing or approving?
- What measurable improvement is expected, against which baseline?
- What would make the organisation stop using it?
Evidence to request: a defined use case, success measures, a process baseline, a named outcome owner and stopping criteria.
2. Agentic behaviour
- Does the system choose its next action, or follow a fixed workflow?
- Can it inspect the result of an action and adapt?
- Can it re-plan when a tool fails or the situation changes?
- What prevents an endless or wasteful loop?
- What conditions force it to stop or escalate?
- Is "multi-agent" a real separation of responsibilities, or several prompts?
Evidence to request: an execution trace, a state diagram, the stopping and retry policies, and a demonstration of one failed case. Parts 3 and 7 of this series explain what good answers look like.
3. Capability and authority
- What is the system technically capable of doing through its tools and accounts?
- What is it authorised to do without approval?
- Which actions are read-only, preparatory, approval-gated or autonomous?
- What financial, operational or risk thresholds apply?
- Which actions are irreversible or difficult to reverse?
- Can the agent expand its own permissions or toolset?
Evidence to request: an authority matrix, the approval policy, the permission model, transaction limits and the rollback design. See Part 4.
4. Tools and integrations
- Which tools, APIs, databases and enterprise systems can it use?
- Are tools narrow business capabilities or broad technical access?
- Can it write to production systems?
- How are tool inputs validated?
- What happens when a tool returns incomplete, stale or contradictory data?
- Does it use MCP, direct APIs, function calling, browser automation or a mixture?
- Who owns each integration and its service level?
Evidence to request: a tool inventory, API contracts, permission scopes, an ownership map, a dependency diagram and failure-mode tests. Our MCP security review lists the questions for tool servers in particular.
5. Data, context, state and memory
- What information is supplied to the agent for each task, and from where?
- How is freshness checked?
- What state is retained during execution, and can a task resume safely after interruption?
- What is remembered across conversations or cases?
- Can users inspect, correct or delete stored memory?
- What prevents information from one customer, case or tenant leaking into another?
Evidence to request: a data-flow diagram, the context sources, the checkpoint schema, a retention schedule, the memory policy and isolation tests. See Part 5.
6. Identity, security and privacy
- Under whose identity does the agent act?
- Can every action be traced to the initiating user, the agent identity and the approving person?
- How are least privilege and separation of duties enforced?
- Where are prompts, retrieved data, model inputs and outputs processed and stored?
- How are secrets and credentials protected?
- How are prompt injection, malicious tool output and data exfiltration addressed?
- What personal or confidential information may the agent process, and under which lawful basis?
Evidence to request: the identity architecture, a threat model, data classification, access reviews, security test results, the incident plan and a privacy assessment. For South African organisations, the privacy assessment should address POPIA's conditions for the specific processing, including any cross-border transfer.
7. Human oversight
- At which exact points can or must a person intervene?
- What evidence does the reviewer receive?
- Does the reviewer have genuine authority to reject or change the action?
- Is there enough time to review, or is approval ceremonial?
- What happens when the person and the agent disagree?
- Who remains accountable for the final outcome?
Evidence to request: the human-decision flow, the review interface, approval logs with timings, the escalation path and a named accountable owner. See Part 6.
8. Reliability and recovery
- What happens when the model, a tool, the network or a downstream system fails?
- How are duplicate actions prevented during retries?
- Can the system resume from a checkpoint?
- How does it recognise that it lacks sufficient information?
- What is the fallback when the agent is unavailable?
- How are partially completed actions reconciled or compensated?
Evidence to request: a failure matrix, the idempotency design, checkpoint and recovery tests, the fallback procedure, a runbook and disaster-recovery expectations.
9. Evaluation and quality
- Against which scenarios has the agent been tested?
- Who defined what "good" looks like?
- Does the test set include normal, ambiguous, adversarial and rare cases?
- How are tool selection, policy compliance and escalation quality measured?
- What are the false-action and missed-action rates?
- How is regression detected when prompts, tools, models or policies change?
- Are automated judges calibrated against expert human judgement?
Evidence to request: the evaluation dataset, the rubric, thresholds, test results, known limitations, regression history and any independent review. Our series Whoever Writes the Rubric Shapes the Agent covers how to judge a rubric.
10. Observability and audit
- Can the organisation reconstruct a complete run?
- Are model, prompt, policy and tool versions recorded with each run?
- Can it show which evidence influenced a recommendation?
- Are cost, latency, retries, errors and escalations observable?
- Can sensitive traces be accessed only by authorised people?
- How long is evidence retained?
Evidence to request: a sample trace, an audit record, dashboards, alert definitions, retention controls and access logs. The Agent Evidence Layer series sets out what a usable evidence trail contains.
11. Economics and value
- What is the full cost per accepted business outcome?
- What costs sit outside the model call: integration, review, exceptions, evaluation, observability and support?
- How much work is rejected, repeated or escalated?
- Does the agent reduce cycle time, improve quality, increase revenue or reduce risk, against the baseline?
- What happens to unit economics as usage scales?
- Which model or tool choices materially affect cost and quality?
Evidence to request: the baseline, a benefits ledger, a full unit-cost model, quality-cost analysis and production usage data. See Cost per accepted outcome, not per prompt.
12. Ownership and lifecycle
- Who owns the business outcome?
- Who owns the agent, its prompts, tools, data, evaluations and runtime?
- Who approves model, prompt, policy and tool changes?
- How are incidents handled, and by whom?
- How are agents versioned, promoted, suspended and retired?
- What happens if the supplier, model or platform must be replaced?
Evidence to request: a RACI, the lifecycle policy, change records, incident ownership, a portability plan and an exit strategy.
A worked example
Ubuntu Municipal Services, the fictional municipal entity used throughout this series, applied the scorecard twice. The first time was to a supplier's "multi-agent complaints platform". The second was to its own service-resolution agent at the end of a three-month pilot.
The supplier's platform scored mostly 0 and 1. Under questioning, the "agents" were a chatbot that collected complaint details and filled a ticket form. That is not a bad product, but it is not what the slides described, and the price assumed agent-level automation. Ubuntu could have bought it as a well-priced intake tool. It could not buy it as an agent.
The pilot agent scored 2 in most domains and 3 in purpose and observability, because it had run on real complaints for three months with traces and a named owner. Its weakest domain was economics: the team could show cost per run but not yet cost per accepted resolution, because acceptance was still measured by hand. That became a condition for scaling, not a reason to stop.
The scores in the example are illustrative.
A twenty-question short form
For an executive meeting where there is no time for the full bank, these twenty questions cover the essentials:
- What outcome does it own, and how will we know it improved?
- Who chooses each next step: a person, the code or the agent?
- What may it change without approval?
- What can it technically change that it is not supposed to?
- Which actions cannot be undone?
- Under whose identity does it act?
- What personal information does it process, and where?
- How does it stop, and what happens when it does?
- What happens when a tool fails mid-action?
- Can it repeat an action by accident?
- Where does a person step in, and what do they see?
- Show me one complete run.
- Show me one failed run.
- What is the test set, and who wrote the rubric?
- What happens when you change the model?
- What does one accepted outcome cost, all in?
- How much work is rejected or escalated?
- Who owns it, and who approves changes?
- How would we switch it off today?
- How would we replace it next year?
Questions to put to your own team
The bank is not only for suppliers. Internal teams are subject to the same optimism, and internal projects often skip the domains that procurement would insist on, especially authority, oversight and lifecycle. Running the bank against an internal agent before it reaches production is cheaper than running it after an incident.
It is also a fair way to treat delivery teams. A team that is asked these questions early can build the evidence as it goes. A team that hears them for the first time at the go-live committee will reasonably feel ambushed.
The CloudNala position
"We have agents" should trigger questions, not applause or dismissal. A credible agent claim states what the agent is trying to achieve, what it can observe, what it can decide, what it can change, when it must stop and what evidence shows it behaved acceptably. Organisations that ask consistently will buy better, build better and avoid paying agent prices for chatbot behaviour, or chatbot attention for agent risk.
This is the final part of the series. Start again at Part 1, Everyone says they have agents. What do they actually have?, or see all parts on the series page.
How CloudNala can help
CloudNala runs agent due-diligence reviews using this question bank. We sit in supplier sessions or internal design reviews, request and assess the evidence for each domain, score it, and give you a written assessment with the conditions to attach to a purchase, a pilot or a production approval.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book an Agent Due-Diligence Review or write to us at consult@cloudnala.co.za