Every quarter at Marula the same argument took place, with different details. The product team wanted to ship the new gift-registry feature before the festive season. The platform lead wanted a two-week freeze, because checkout had been flaky in October. Both had a case. Neither had a number. The argument was settled, as it usually is, by whoever the managing director spoke to last.
What the meeting lacked was not goodwill. It was a shared, measured answer to two questions: how reliable does checkout need to be, and how reliable has it actually been? Service level objectives and error budgets exist to provide exactly that.
Marula Commerce is fictional, and its systems and figures are illustrative. Marula is the Pretoria online homeware retailer this series has followed. Its checkout depends on the shop platform, the catalogue service, a payment provider and a courier quote API, and it sells directly and through two marketplace partners.
Three terms, one idea
The terms come from site reliability engineering, and Google's book on the subject defines them carefully in its chapter on Service Level Objectives.
A service level indicator (SLI) is "a carefully defined quantitative measure of some aspect of the level of service that is provided". In plain English: a number that describes what the customer experiences. The share of checkout attempts that succeed is an SLI. So is the share of product pages that load within two seconds.
A service level objective (SLO) is "a target value or range of values for a service level that is measured by an SLI". In plain English: the level you aim for. "99.5% of checkout attempts succeed, measured over 28 days" is an SLO.
A service level agreement (SLA) is "an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain". In plain English: a promise to someone else, with a penalty attached. Marula's contracts with its marketplace partners include availability commitments for the product feed. Those are SLAs.
What does production-ready actually mean? lists an agreed objective with a measured history as the first area of readiness evidence. This part explains how to set one. The order matters. You choose what to measure, then set an internal target, then, if you must, make a contractual promise that is looser than the internal target, so that you notice trouble before it costs you money.
Measure what the customer feels
The most common mistake with SLIs is to measure what is easy rather than what matters. Server uptime, CPU and memory are easy. A customer does not experience any of them directly. A customer experiences a checkout that works or does not, and a page that loads quickly or does not.
Part 4 made the same argument about alerts: page on what the customer feels, not on resource metrics. SLIs are where that principle is written down. Marula chose four:
| Journey | SLI | How it is measured |
|---|---|---|
| Checkout | Share of checkout attempts that complete or fail for a customer reason, such as a declined card | Gateway and application logs; declines excluded as valid outcomes |
| Product browsing | Share of product page requests served within two seconds | Gateway latency measurements |
| Order status | Share of status lookups that return current data | Application metric with a freshness check |
| Partner feed | Share of hourly feed deliveries to marketplace partners that arrive complete | Feed job records |
Each SLI is a ratio of good events to all events. That form is easier to reason about than averages, and it turns directly into an error budget.
Why not one hundred per cent?
It is tempting to set every objective at one hundred per cent. The SRE book is direct about why that is wrong: "100% availability is impossible", and insisting that SLOs are met all the time is "both unrealistic and undesirable".
It is impossible because every component fails sometimes, including the payment provider, the courier API, the cloud platform and the customer's own mobile connection. It is undesirable because the cost of each additional nine rises steeply, and because a team that may never fail can never change anything. A system that is not allowed to fail is a system that is not allowed to improve.
The right objective is the level below which customers notice and care. For Marula's checkout, the operations director and the product owner looked at past complaints and abandoned-basket data and agreed on 99.5% of checkout attempts over a rolling 28 days.
The error budget is the gap
If the objective is 99.5%, then 0.5% of checkout attempts are allowed to fail. That allowance is the error budget.
In a 28-day window Marula handles about 200,000 checkout attempts. Half a per cent of that is 1,000 failed attempts. That is the budget. Every failure spends some of it, whatever the cause: a bad release, a payment provider outage, a database failover.
The budget changes the quarterly argument. The question is no longer "is checkout reliable enough?", which has no answer. It is "how much of the budget is left?", which does. If 300 failures have been spent with three weeks to go, there is room to ship the gift registry with care. If 950 have been spent, there is not, and everybody can see why.
The budget also cuts the other way. If the budget is consistently barely used, the objective may be too loose, or the team may be too cautious with releases. An unused budget is not an achievement. It may be a sign that the business is paying for reliability nobody asked for, or giving up delivery it could have had.
Write the policy before you need it
An error budget is only useful if everyone has agreed, in advance, what happens when it runs low. Google's SRE workbook includes an example error budget policy in which a service that exhausts its budget halts "all changes and releases other than P0 issues or security fixes" until it is back within its objective. It also states that the policy "is not intended to serve as a punishment for missing SLOs".
Marula's policy is shorter and fits its size:
- Below half the budget spent: normal releases.
- Half spent: the platform lead reviews the causes with the product owner. Releases continue, with extra care for changes that touch checkout.
- Three quarters spent: releases that touch checkout need the platform lead's approval and a rollback plan tested in the last week.
- Budget exhausted: feature releases to checkout stop, except fixes for the causes of failure and security fixes, until the rolling window is back within the objective. Engineers move to reliability work.
- Exception: the managing director may override the policy for a specific release, in writing, with the reason recorded.
The policy is signed by the product owner and the platform lead, not imposed by one on the other. That is what turns reliability from an argument into a shared rule.
Dependencies set a ceiling
An SLO cannot be higher than the reliability of what it depends on, unless the design compensates. Marula's checkout calls the courier quote API. If that API succeeds 99% of the time, and checkout fails whenever it fails, checkout cannot reach 99.5%.
There are three honest responses. Negotiate a better commitment from the dependency. Design around it, for example by showing a standard delivery fee when the quote API does not answer in time, so checkout completes, with the timeout and retry limits from Part 5 deciding when "in time" has passed. Or set an objective the dependencies can actually support. Marula chose the second, and checkout now completes with a standard fee when the courier API is slow. The courier failure no longer spends checkout's budget.
The same reasoning applies to AI features. If a product-search assistant depends on a hosted model, the provider's availability and latency set its ceiling, and the design needs a fallback, such as conventional search, for when the model is slow or unavailable. Where checkout spans several services, a successful payment can still end in a failed order, and the indicator should count that outcome as a failure.
What happened to the festive release
In the first 28-day window with the new policy, checkout had spent about 80% of its budget by the planned release date, most of it during a single payment provider incident. That was past the three-quarter mark, and the gift registry touched checkout, so under the policy it needed the platform lead's approval and a tested rollback. It got both and shipped a week later than the product team wanted, behind a feature flag. The quarterly argument did not happen, because the numbers had already answered it.
Questions to ask about any reliability target
- Is each objective based on an indicator the customer actually experiences?
- Is each indicator a ratio of good events to all events over a stated window?
- Who agreed the objective, and on what evidence of what customers notice?
- Is any contractual commitment looser than the internal objective?
- Is there a written budget policy, agreed in advance by both product and engineering?
- Which dependencies cap the objective, and how does the design handle them?
- Is the budget ever left unused for long periods, and if so, why?
Key points
- An SLI measures what the customer experiences, an SLO sets the internal target, and an SLA is a contractual promise with consequences.
- One hundred per cent is the wrong target: it is impossible to meet and prevents change.
- The error budget is the allowed shortfall, and it turns "is it reliable enough?" into "how much budget is left?".
- Agree an error budget policy before the budget runs low, signed by both product and engineering.
- Dependencies set a ceiling on any objective unless the design degrades gracefully around them.
This is the final part of Architecture in Plain English. Start again from Part 1, What an API gateway actually does, and see Part 9, A backup is not restore confidence for the recovery side of reliability.
How CloudNala can help
CloudNala helps product and engineering teams set reliability targets they can both stand behind. We identify the customer journeys that matter, define indicators from your existing telemetry, facilitate the conversation that sets each objective, and draft an error budget policy with the escalation and exception rules your organisation will actually use.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request an Architecture Review or write to us at consult@cloudnala.co.za