Observability is often sold as three pillars: logs, metrics and traces. The phrase is tidy and slightly misleading. It suggests three equal things you should have, when in practice they are three different tools for three different questions, and an incident goes well or badly depending on whether the team reaches for the right one at the right moment.
This part explains each signal by the question it answers, then follows one incident from the first alert to the fix.
Marula Commerce is fictional, and its systems and figures are illustrative. It is a Friday evening during a promotion. Marula's checkout, which now calls a separate catalogue service and an external courier API for delivery quotes (see Part 2), starts failing for some shoppers.
Three questions, three signals
Metrics answer "is something wrong, and how much?" A metric is a number measured over time: requests per second, error rate, response time at the 95th percentile, queue depth. Metrics are cheap to store, fast to query and ideal for alerts. They tell you that checkout errors rose from 0.4% to 6% at 19:42. They cannot tell you why.
Traces answer "where in the path is it wrong?" A trace follows one request through every component it touches, recording each step as a span with a start time and duration. In a system with several services, a trace is the only signal that shows the whole journey of a single checkout. It tells you that the slow part is a nine-second call to the courier quote API, not the database and not the catalogue.
Logs answer "why did this particular request fail?" A log is a timestamped record of something that happened, written by the code. Good logs carry the detail that neither metrics nor traces hold: the error message, the input that triggered it, the retry count. They tell you "courier quote timeout, retry 3 of 3, falling back to none".
Each signal is weak alone. Metrics without traces leave you guessing which service is responsible. Traces without logs show you where time went but not what the code decided. Logs without metrics give you thousands of lines and no sense of scale.
Following the incident
- The alert fires. A rule on the checkout error-rate metric pages the on-call engineer when errors exceed 2% for five minutes. Good alerts are on symptoms customers feel, not on CPU or memory.
- The dashboard narrows it. The checkout dashboard shows errors concentrated on the "get delivery options" endpoint. Response time on that endpoint has jumped from 300 milliseconds to over 9 seconds.
- A trace sample locates it. Opening a few slow traces shows the same span every time: the outbound call to the courier quote API.
- Logs explain it. Filtering logs by the trace ID from one of those traces shows three timed-out attempts, each waiting three seconds, and then a failure that the checkout code turned into a generic error instead of offering standard delivery.
- Fix and confirm. The team reduces the courier timeout, stops retrying three times inside a user request, and shows a default delivery option when quotes are unavailable. The error-rate metric drops back below 1%, which is how they know the fix worked.
The thread connecting steps three and four is the trace ID. When every log line includes the ID of the trace it belongs to, moving from "this request was slow" to "here is what the code did" takes seconds. Without it, the engineer searches logs by timestamp and hopes. The industry standard for passing that ID between services is the W3C Trace Context traceparent header, which most current instrumentation libraries, including OpenTelemetry, propagate automatically.
What to put in place before the next incident
| Item | Why it matters |
|---|---|
| Alerts on customer-facing symptoms | Error rate and latency on key journeys, not server resources |
| One dashboard per critical journey | Checkout, sign-in, payment: the first place on-call looks |
| Tracing across every service hop | Without it, multi-service incidents are guesswork |
| Trace ID in every log line | The link between "where" and "why" |
| Structured logs | Fields that can be filtered, not free text that has to be read |
| Retention matched to need | Metrics for months, traces and detailed logs for days or weeks |
| A cost owner | Observability data grows quietly and can rival hosting costs |
The last row is worth stressing. High-volume logging and full trace capture are expensive. Most teams sample traces (keeping all errors and a percentage of successes) and set retention by signal. Deciding this deliberately is cheaper than discovering the bill.
Four habits that make incidents longer
Logging everything at the same level. When every line is "info", the one line that matters is buried. Agree what each level means: errors are things someone should act on, warnings are degraded but handled, and debug detail is off in production unless switched on for a specific investigation.
Averages instead of percentiles. An average response time of 400 milliseconds can hide the fact that one shopper in twenty waits nine seconds. Alert and report on the 95th or 99th percentile for anything customer-facing.
Dashboards without owners. A dashboard built for a launch and never revisited shows metrics nobody understands, with thresholds nobody remembers setting. Each critical-journey dashboard should have a named owner who reviews it after every incident.
Alerts that page for causes, not symptoms. Paging someone because CPU reached 80% trains the team to ignore pages. Page on what the customer feels, such as failed checkouts or slow sign-in, and keep resource metrics on dashboards for diagnosis.
Logs and personal information
Logs often capture more than intended: email addresses, phone numbers, delivery addresses, sometimes full payment payloads. Under POPIA that makes your log store a store of personal information, with the same security and retention obligations as the database. Mask or omit personal fields at the point of logging, restrict who can search logs, and set retention to what operations genuinely need.
When the system includes AI
AI components add a fourth question: "was the answer any good?". A trace can show that a model call took two seconds and cost a certain number of tokens. It cannot show whether the answer was correct. That needs evaluation, which is a separate discipline. We cover it in Observability and cost: why production AI must be visible and A trace is not an evaluation.
The basics still come first. A team that cannot follow an ordinary checkout request through its services will not be able to follow an agent through its tool calls.
Next: Part 5, HTTP status codes as operational signals. All parts are on the series page.
How CloudNala can help
We help teams decide what to measure on their critical journeys, put tracing and correlated logging in place across service boundaries, and set retention and sampling so that observability stays affordable. We also run incident walk-throughs that test whether the signals actually answer the questions on-call will ask.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request an Architecture Review or write to us at consult@cloudnala.co.za