For three years, Marula's backup report arrived every morning and every line was green. The database was copied nightly, transaction logs every fifteen minutes, and the copies were kept for thirty days in a separate storage account. When the finance director asked, during an insurance renewal, whether Marula could recover from losing its main database, the answer was an easy yes.
Then a developer ran a data migration against production instead of the test environment and deleted most of the discount codes for a promotion that started at midnight. The backups were there. Restoring them was another matter. The only documented procedure restored the whole database, which would have rolled back every order taken that afternoon. Nobody had restored a single table before. It took until four in the morning to recover the codes into a scratch copy and move them back by hand, and the promotion opened with half its codes missing.
Nothing about the backups had failed. What had never been tested was the restore.
Marula Commerce is fictional, and its systems and figures are illustrative. Marula is the Pretoria online homeware retailer this series has followed. After the catalogue extraction in Part 2, it runs a shop platform with its own database, a catalogue service with another, product images in object storage, and integrations with a payment provider, a courier and two marketplace partners.
The plain-English definition
A backup is a copy of data taken at a point in time. Restore is the act of using that copy to put data back. Recovery is getting the service working again for customers, which needs restored data plus everything else the service depends on.
Most organisations measure the first and assume the other two. Google's site reliability engineering book puts it bluntly in its chapter on data integrity: "No one really wants to make backups; what people really want are restores." The same chapter's advice is to "prove that data recovery works with regular exercise, or data recovery won't work" (Data Integrity: What You Read Is What You Write).
What does production-ready actually mean? already lists a rehearsed restore as one of nine areas of readiness evidence. This part explains what that rehearsal should prove, and why a single successful test is not the end of it.
Two numbers the business must choose
Every recovery plan rests on two numbers. They are business decisions, not technical ones, and they are usually never made explicitly.
The recovery point objective (RPO) is how much data the business can afford to lose, measured in time. If Marula's RPO for orders is fifteen minutes, then after a failure it must be able to recover every order placed up to fifteen minutes before the failure. Anything placed in those last fifteen minutes may have to be re-entered or reconciled from the payment provider.
The recovery time objective (RTO) is how long the service can be unavailable. If the RTO for checkout is four hours, the shop must be taking orders again within four hours of the failure.
Shorter numbers cost more. A fifteen-minute RPO needs frequent log backups; a near-zero RPO needs replication to a second database. A one-hour RTO needs a rehearsed procedure, automation and probably standby infrastructure. The honest conversation is about which parts of the business need which numbers.
Marula's answers, after a two-hour session with the operations director:
| Data | RPO | RTO | Why |
|---|---|---|---|
| Orders and payments | 15 minutes | 4 hours | Money and customer promises; payments can be reconciled from the provider for the gap |
| Discount codes and promotions | 1 hour | 2 hours during a promotion | Small data, high timing pressure |
| Product catalogue | 24 hours | 8 hours | Changes slowly; can be re-imported from suppliers |
| Product images | 7 days | 24 hours | Can be re-uploaded from the design team's originals |
Notice that the discount codes needed a shorter RTO than the database as a whole. A single recovery procedure for everything could never have met it.
What a green backup does not tell you
A backup job reports success when it has written a copy. It does not check most of what a restore depends on. In restore drills, problems like these are common:
- The copy is incomplete. A table, a schema or a storage container was added after the backup job was configured and was never included.
- The keys are missing. The data is encrypted and the key, or the secret needed to reach the key store, is not available in the recovery environment.
- The order is wrong. Services must be restored in a particular sequence, and nobody wrote it down.
- The parts disagree. Two databases restored to different points in time hold inconsistent data. Marula's shop database might show an order for a product that the restored catalogue does not contain. That is the same consistency problem as a successful payment and a failed order, created by recovery instead of by a crash.
- It takes too long. The restore works, but copying the data back takes longer than the RTO allows.
- Nobody knows how. The procedure lives in the memory of one engineer, who is on leave.
None of these show up on a backup dashboard. All of them show up in a drill.
A ladder of restore confidence
Restore confidence is not a yes or a no. It is a level of evidence, and each level is worth more than the one below it.
- Backup completed. The job reports success. Necessary, and nearly worthless on its own.
- Backup readable. An automated check opens the copy and confirms it is complete and not corrupted.
- Restored in isolation. The copy is restored into a separate environment, and the time taken is recorded.
- Data verified. Business checks confirm the restored data is right: the latest orders are present, record counts match, totals reconcile with the payment provider's report for the period.
- Service works. The application runs against the restored data, and someone can place a test order.
- Recovery rehearsed. The full scenario, including restore order, keys, configuration and integrations, has been rehearsed within the RTO by someone other than the person who wrote the procedure.
Most organisations are at level one or two. Level four is where restore confidence begins, because it is the first level at which the data has been checked by someone who knows what it should contain. Level six is the evidence an auditor, insurer or board should ask for.
Different data can sit at different levels. Marula deliberately left its product images at level three, because they can be re-uploaded from the design team's originals and the cost of going further is not justified. Orders need level six.
Rehearse more than one kind of failure
A full disaster, where the whole database or region is lost, is the scenario that recovery plans usually describe. It is not the most common one. Marula's own history suggests three scenarios worth rehearsing:
- Accidental change. A bad migration, a mistaken bulk update, an integration that overwrote good data. This needs point-in-time and partial restores: one table or a set of records to a scratch environment, then a careful merge back.
- Loss of a component. A database server or storage account fails. This needs a whole-component restore and a check that the other components still agree with it.
- Loss of access. Credentials are compromised or data is encrypted by an attacker. This needs copies the attacker cannot reach or alter, which is why recovery copies should sit under separate credentials and, where the platform supports it, be protected from deletion for a set period. Verify what your cloud platform actually offers before relying on it.
Each scenario has a different procedure and a different time. Rehearsing only the dramatic one leaves the everyday ones to improvisation, which is what happened with the discount codes.
Make the drill routine
Restore confidence decays. Every new table, service, key rotation or change of team member can break a procedure that worked last quarter. Marula now runs restore drills on a schedule tied to each data set's importance: monthly automated restores with business checks for orders, quarterly partial-restore drills for promotions before major campaigns, and an annual full recovery rehearsal.
Each drill produces a short record: the scenario, the time taken against the RTO, the data verified against the RPO, the problems found and who will fix them. That record belongs in the living architecture documentation next to the diagram of what is being recovered, because a recovery procedure is part of the architecture, not an operations afterthought. Drill failures are tracked like any other defect.
The first full drill took Marula nine hours against a four-hour RTO. Two problems explained most of the gap: the catalogue service's database was restored after the shop's, which left the shop unable to show products, and the encryption key for the images container lived only in the production key vault. Both were fixed in a fortnight. The second drill took three and a half hours.
Questions to ask about any backup
- What are the RPO and RTO for each important data set, and who in the business agreed them?
- Which level of the restore confidence ladder has each data set reached, and when?
- Can you restore a single table or set of records without rolling back everything else?
- If several components are restored, how do you make them consistent with each other?
- Are keys, secrets and configuration recoverable in the same scenario as the data?
- Could an attacker with production credentials delete or alter the recovery copies?
- Who other than the author has run the procedure, and how long did it take?
Key points
- A backup is a copy; a restore puts data back; recovery gets customers served again. Measure all three, not only the first.
- Recovery point and recovery time objectives are business decisions, set per data set, not one number for everything.
- Most restore problems, such as missing data, missing keys, wrong order and inconsistent components, are invisible until a drill.
- Treat restore confidence as a ladder of evidence, and aim for verified data and a working service, not a completed job.
- Rehearse accidental change, component loss and loss of access, on a schedule that matches each data set's importance.
Next: Part 10, SLOs and error budgets in plain English. All parts are on the series page. For what to watch while recovering, see Logs, metrics and traces when production breaks.
How CloudNala can help
CloudNala runs recovery reviews for cloud workloads. We help the business set recovery objectives for each data set, map what a restore depends on, design and run the first drills with your team, and leave a schedule and drill record that keep restore confidence current as the architecture changes.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request an Architecture Review or write to us at consult@cloudnala.co.za