CloudNala Builds19 August 20268 min read

From Chatbot to Agent — Part 13 of 15

Building an internal operations agent before you sell one

The most honest way to learn what agentic AI is actually like to run is to point one at your own operations, where the consequences of getting it wrong land on you.

#CloudNala Builds#Agentic AI#Operations#AI Strategy

There is a credibility problem in AI consulting at the moment, and it is worth naming plainly: a great many organisations selling agentic AI have never operated one where a mistake costs them something.

The remedy is straightforward. Build one for your own business first, in a workflow you genuinely rely on, and live with it for a while.

The problem we pointed it at

Opportunity capture. A consulting business generates leads from an unhelpful variety of sources: an email, a conversation on a messaging app, a screenshot someone forwards, a note after a meeting, a mention in a conversation that goes nowhere for three months and then matters.

The failure mode is not that opportunities are rejected. It is that they evaporate. Nobody wrote it down, the follow-up did not happen, and six weeks later somebody remembers vaguely that there was a conversation with someone about something.

This is a workflow where the mechanical steps genuinely dominate. Extracting who and what from an unstructured input, matching it against what we already know, classifying it against the service catalogue, deciding what the next action is, drafting it, and recording it — most of that is preparation, and preparation is exactly what a person defers when they are busy.

The shape of it

flowchart TD
    I[Email, message, screenshot or meeting note] --> E[Extract organisation, contact, context]
    E --> M{Do we already know them?}
    M -- Yes --> H[Attach to the existing record and history]
    M -- No --> N[Create a new record, flagged as unverified]
    H --> C[Classify against the service catalogue]
    N --> C
    C --> S[Assess urgency and fit]
    S --> D[Draft the next action and the follow-up message]
    D --> R[Route to a human with the reasoning attached]
    R --> A{Approved?}
    A -- Yes --> U[Send and update the record]
    A -- No --> F[Human edits — the edit is captured as a signal]

The two decision diamonds are where the design work went.

The first — do we already know them — is where the system either builds institutional memory or corrupts it. Matching too eagerly merges two different organisations with similar names. Matching too cautiously creates duplicates and loses the history. We resolved it by matching conservatively and flagging uncertain matches for a person, on the basis that a duplicate is annoying and a bad merge is expensive to unpick.

Matching a new contact against what you already knowA new contact arrivesConfident match:attach to the existingrecord and historyUncertain: flag itfor a person ratherthan guessingNo match: create anew record, markedunverifiedA duplicate is annoying. A bad merge is expensive to unpick. Bias towards the annoying one.
Early versions of this were too willing to associate. The results looked good in aggregate and were wrong in exactly the cases that mattered — so the middle path became the default whenever confidence was not high.

The second — approval — is where the whole thing stays safe. Nothing is sent under the company's name without a person approving it. Not because the drafts are poor, but because a message to a prospective client is a commitment, and the asymmetry described in the article on tools applies with full force.

Four things we learned

Extraction was harder than reasoning. The reasoning step — classify this, assess fit — worked well almost immediately. Turning a screenshot of a conversation into reliable structured fields took considerably longer. This inverts the assumption most people bring, and it matches what we now see on client engagements: perception is the expensive layer, and it is the one nobody budgets for.

The human edit is the best quality signal available. When a reviewer consistently rewrote the same part of a draft, that identified a specific weakness far more precisely than any subjective assessment. Capturing the edit — not just that it happened, but what changed — turned quality improvement into something concrete. It is free training data that nobody has to be asked to produce.

Conservative matching beat clever matching. Early on the system was too willing to associate a new contact with an existing organisation. The results looked good in aggregate and were wrong in the specific cases that mattered. Lowering confidence and escalating ambiguity produced a system people trusted more, even though it did slightly less work.

Stateless was enough for longer than expected. We deferred long-term memory entirely at first, and the workflow functioned. What that period revealed was exactly which facts the system kept needing — and those facts became a curated, human-owned knowledge set rather than anything the agent wrote for itself. This is the sequence we now recommend in the article on memory, and we recommend it because deferring memory cost us nothing and taught us what to build.

What it does not do

It does not send anything without approval. It does not score or rank people. It does not store personal information beyond what is needed to contact someone about a business matter, and what it does store is subject to the same retention rules as the rest of our records. It does not make commercial decisions.

Being specific about the limits matters more than describing the capabilities, because the limits are the part that transfers. Any organisation building something similar will face the same question about where the approval sits, and the answer we would give is: one step before anything leaves the building.

Why this is worth doing before selling it

Three reasons, and the third is the important one.

The first is competence. Running an agent through model updates, cost surprises, extraction failures and awkward edge cases teaches things that reading cannot.

The second is honesty in the sales conversation. Being able to say what actually took longer than expected, and what we would do differently, is more useful to a client than a capability deck.

The third is that it forces the discipline. It is easy to recommend evaluation suites, cost ceilings, audit trails and approval gates to somebody else. It is a different exercise to build them for yourself, on your own budget, and discover which ones you were tempted to skip. Every recommendation in this series is one we have had to apply internally, and several of them are in the series precisely because we initially skipped them.

How to start small

If you want to try this in your own organisation, choose a workflow where you are the customer. Something where the output goes to you or your colleagues, where a mistake is embarrassing rather than expensive, and where you already know what good looks like.

Instrument the current version. Build the extraction step only. Let a person do everything else for a month. Then add one more step.

You will learn more from that than from a vendor evaluation, and you will end up with a considerably better brief for whatever you build next.

How CloudNala can help

We work with organisations to identify the internal workflow worth automating first — usually one with high mechanical content, a forgiving failure mode and a clear owner — and build it with the approval gates, traces and evaluation in place from the start. Starting internally is the fastest way to build organisational confidence with AI, because the lessons land where they are cheapest.


Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Readiness Workshop or write to us at consult@cloudnala.co.za