Measure avoided demand, not conversations

The vendor dashboard looked excellent. In its first month the assistant at Bontebok Furniture Group, a fictional South African furniture retailer with stores and home delivery, had handled about 600 conversations, closed most of them without a person, and scored well on a thumbs-up survey. The vendor's account manager suggested a case study.

Fatima, the data analyst who had counted contacts since the start of this series, asked a different question: did customers contact Bontebok less? The dashboard could not say. It counted what happened inside the assistant. It did not count the calls, the store visits, the emails or the WhatsApp messages that went to people. It did not know how many orders had been delivered. And it could not say what would have happened without the assistant.

Bontebok's people, systems and figures in this series are illustrative. The measurement problem is not. Most AI service dashboards measure the activity of the new channel. The outcome this series has pursued since part 1 is different: customers should not need to contact Bontebok to find out about their delivery. That is measured in demand avoided, not in conversations handled.

Why conversation counts mislead

Conversation counts rise for good and bad reasons, and the dashboard cannot tell them apart.

• Displacement. A customer who would have phoned now chats instead. The conversation count rises, total contacts do not change, and the dashboard reports success.
• Induced demand. Making contact easier invites more of it. A customer who would have waited for the delivery now asks twice.
• Repeat demand. A customer whose conversation "closed" without a person, but who did not get what they needed, comes back through another channel. The assistant records a success; another channel records a new contact.

Our Production AI Case Study already argues that deflection is the wrong target for an assistant and replaces it with resolution without a repeat contact. That is the right measure for whether a conversation worked. This part adds the measure one level up: whether the demand should have existed at all, and whether it went down.

Failure demand and value demand

The distinction that makes demand measurable comes from John Seddon, who writes on the design of service organisations. In Freedom from Command and Control (2003) he set out failure demand as demand caused by a failure to do something or do something right for the customer, and contrasted it with value demand, the demand a service exists to meet.

At Bontebok the line was clear once it was drawn:

• Value demand: a customer choosing a delivery slot, giving access instructions, asking how to care for a leather couch, buying a mattress protector. These are reasons the service exists. Making them easier is good.
• Failure demand: "where is my order?", "why did nobody tell me the date moved?", "the driver did not come", "only half my bed arrived". These exist because something upstream failed. The aim is to remove them, not to answer them more cheaply.

Fatima added this classification to the reason codes she had used in part 1. Every sampled contact was coded as value or failure demand, with the cause. That made one fact unavoidable: an assistant that answered "where is my order?" brilliantly would be optimising failure demand. Fixing the information gap removed it.

Defining the measure

The measure Bontebok adopted was simple to state and took some work to collect.

Three details in that table matter more than they appear to.

An assistant conversation is a contact. It goes in the numerator like any other. Leaving it out is the most common way an assistant appears to reduce demand that it has only moved.

Proactive messages are not contacts. A message Bontebok sends is part of the fix, not demand. If customers reply, the reply is a contact.

The denominator is the business, not the channel. Dividing by orders due for delivery stops a busy month from looking like a failure or a quiet month from looking like a success. Bontebok's order volumes have strong seasonal peaks, and a raw count would have swung with them.

The instrumentation behind the numerator draws on the approach in our article on instrumenting the customer journey, not only the infrastructure: every contact in every channel carries the order key, so it can be joined to the delivery it was about.

A comparison group, not a before-and-after chart

A before-and-after chart is the weakest evidence of change, because many things change at once: the season, a new carrier, a supplier who suddenly delivers on time. Bontebok kept a comparison region from the start.

Illustratively, total delivery contacts (failure plus value demand) per 100 orders due, over the six-week pilot from part 1:

Failure demand was tracked alongside: in the pilot it fell from 58 to 22 per 100 orders due, in the comparison region from 57 to 55, so the failure-demand difference is also about 34. Value demand stayed at about 9 in both regions, which is why the totals and the failure-demand figures tell the same story here. The comparison region's small fall is the background change that would probably have happened anyway; subtracting it gives a better estimate of what the intervention did. This is a simple difference-in-differences comparison. It is not a randomised trial, and it assumes the two regions would otherwise have moved in parallel, which is why the team checked that their trends had matched in the months before the pilot.

With about 3,500 orders due a month in the pilot region, a reduction of about 34 contacts per 100 orders due is roughly 1,200 avoided contacts a month. That is the number that went to the steering committee, with the method beside it.

What happened when the assistant arrived

The assistant from part 4 launched in the pilot region two months later. The vendor dashboard reported its conversations. Fatima's measure reported something more useful.

Illustratively, total delivery contacts in the pilot region moved from 31 to 27 per 100 orders due, including the assistant's own conversations. Failure demand fell from 22 to 16: fewer repeat calls about missed slots, and fewer customers phoning after failing to reach a coordinator. Value demand rose from 9 to 11: more customers chose a slot and gave access instructions, which was the intended effect and reduced failed deliveries. In the comparison region, nothing changed.

That is a modest result, and an honest one. The assistant added a little to an improvement that the information fix had already made. Had the total risen after launch, the measure would have shown it, and the team would have looked for displaced or induced demand before celebrating.

Turning avoided contacts into money, carefully

Avoided contacts are an outcome. They become a financial benefit only if the capacity they free is used or released. Bontebok's contact centre did not reduce headcount. It moved two agents onto outbound calls for failed deliveries, which shortened the delay that part 3 had found between a failed delivery and a rebooked slot.

That is a real benefit, but it is not the same as a cost saving, and it should not be reported as one. Our articles on who tracks the benefits after a business case is approved and why AI consumption is not AI value cover how to keep that distinction honest in a benefits register.

Pitfalls to watch

• Coding drift. If the people classifying contacts change their interpretation, the trend changes without any real change. Re-code a sample each month and keep the definitions written down.
• Missing channels. Store visits are easy to forget and expensive to count. A two-week tally each quarter is better than leaving them out.
• Suppressed demand. Demand can fall because contact became harder, not because the cause was fixed. Watch complaints and abandoned calls alongside the measure.
• Mixing value and failure demand. A target to reduce all contacts will eventually push teams to discourage legitimate ones. Target failure demand; report value demand.
• Vendor metrics as outcomes. Conversation counts, containment and survey scores describe a channel. Keep them, but do not put them in the benefits register.

Key points

• Conversation counts measure a channel's activity. They cannot show whether customers needed to contact you less.
• Separate failure demand, which should be removed, from value demand, which a service exists to meet.
• Count every contact in every channel, including assistant conversations, and divide by the volume of underlying business.
• Use a comparison group, and check that it tracked the pilot before the change.
• Report avoided contacts as an outcome; claim a financial benefit only when the freed capacity is used or released.

Next: Part 6, From discovery evidence to architecture decision. All parts are on the series page.

How CloudNala can help

CloudNala helps service teams measure what a digital or AI change actually avoided. We define failure and value demand for your service, set up contact classification across every channel, join contacts to the underlying business volume, design a fair comparison group and report the result with its method, so that your steering committee sees avoided demand rather than a vendor's conversation count.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Start a Digital Discovery Sprint or write to us at consult@cloudnala.co.za