Observability

Logs, metrics and traces, what to measure, and how SLOs and error budgets decide when to page someone.

Draft. This page is new and has not yet been reviewed by a second person. If something looks wrong, report it.

On this page
  1. Three signals
  2. Logs
  3. Metrics
  4. What to measure
  5. Traces
  6. SLIs, SLOs and SLAs
  7. Error budgets
  8. Alerting
  9. In an interview
  10. Sources and further reading

Observability is being able to understand what a running system is doing from the outside, using what it emits, including for problems nobody predicted. Monitoring is the part that watches for known conditions and tells you when one occurs. A design that scales but cannot be debugged at 3 a.m. is unfinished, and interviewers often ask how you would know your system is healthy, and what you would look at when it is not.

Three signals#

Systems emit three kinds of data. Each answers a different question, so you need all three, and it helps a great deal if they can be linked.

SignalWhat it isThe question it answersIts weakness
Metricsnumbers measured over timeIs something wrong, and how much?no detail about any one request
Tracesthe path of one request through the servicesWhere did this request spend its time?usually only a sample is kept
Logstimestamped records of eventsWhat exactly happened here?expensive at volume, and no overview

A typical investigation moves through them in that order: an alert fires on a metric, a trace shows which service is slow, and the logs for that trace show why.

Logs#

Prefer structured logs, with consistent named fields, over free text. A consistent schema makes logs easy to parse and analyse at scale, and to link with traces:

json
{"time": "2026-11-05T09:30:00Z", "level": "error", "service": "checkout", "msg": "charge failed", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "order_id": "A1042"}

The field that does the most work is the trace ID: it turns "all the logs" into "the logs for this one request". Keep secrets and personal data out of logs, and remember that logging every request at high traffic is a real cost.

Metrics#

Most metrics are one of three kinds:

  • A counter only goes up, like total requests. Its rate of change is what you graph.
  • A gauge is a current value, like queue length or memory in use.
  • A histogram counts values into buckets, like request duration. This is how you get percentiles.

Two mistakes are common.

Trusting the average. Averages hide what users feel. The Google SRE book's example is a service averaging 100 ms at 1,000 requests a second, where 1% of requests might easily take 5 seconds. That is 10 requests every second, and the rest would be averaging about 50 ms, which looks fine on a graph. Look at percentiles such as the 99th. The SRE book suggests counting requests into latency buckets with roughly exponential boundaries.

The tail also grows with fan-out. If one server is slow on 1% of requests, a request that needs 100 such servers in parallel is slow 63% of the time, as The Tail at Scale shows. The slowest response decides. The same paper describes one fix, hedged requests: send a second copy after a short delay and use whichever answers first.

Averaging percentiles, or adding too many labels. Percentiles computed on each instance cannot be combined: the Prometheus documentation warns that averaging them gives statistically nonsensical values. Sum histogram buckets across instances and compute the percentile from the total. An estimate is only as good as its buckets, though: Prometheus's example has a true value of 220 ms reported as 295 ms from a 200 to 300 ms bucket.

Every unique combination of label values is a new time series, so the count multiplies. 5 methods, 20 routes and 5 status codes make 500 series, which is fine. Add a user ID label with a million users and it is 500 million. Prometheus's guidance is not to use labels for unbounded values such as user IDs or email addresses. Put those in logs and traces instead.

What to measure#

You do not need to invent a list. Three short ones are widely used:

MethodForMeasure
The four golden signalsany user-facing systemlatency, traffic, errors, saturation
REDrequest-driven servicesrate, errors, duration
USEresources such as CPUs and disksutilisation, saturation, errors

Some details from the SRE book are worth repeating in an interview. Track the latency of successes and failures separately, since a fast error can flatter your average. Count requests that return a success with wrong content, or that are too slow to be useful, as errors. And saturation matters before 100%, since performance often degrades earlier, as the load curve lab shows.

Traces#

A trace records the path of one request. It is made of spans, each a unit of work with a start and an end, such as a service call or a database query. Every span in a trace shares a trace ID, and each records its parent, which gives a tree:

text
GET /checkout                       root span: the whole request
├─ auth-service: verify token
├─ cart-service: load cart
│  └─ database: SELECT items
└─ payment-service: charge
   └─ card network: authorise

For this to work across services, the IDs must travel with the request. This is context propagation. The standard is the W3C traceparent header, which carries a version, a 16-byte trace ID, an 8-byte parent ID and flags, including whether the trace is sampled:

text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

One service that fails to pass the header on breaks the trace in two, which is why tracing is usually built into shared libraries. Google's Dapper paper names restricting instrumentation to a few common libraries as one of the choices key to its success, along with sampling.

Sampling keeps the cost down, and the OpenTelemetry documentation says a rate of 1% or lower often suffices at high volume. There are two ways to decide:

  • Head sampling decides at the start of the request. It is simple and cheap, but it cannot know yet whether this request will fail or be slow.
  • Tail sampling decides after seeing most of the trace, so it can keep the errors and the slow ones. It needs stateful components that use real resources and are hard to run.

SLIs, SLOs and SLAs#

These three terms are often confused, and interviewers notice.

TermWhat it isExample
SLI, indicatora measurement of the servicethe share of requests answered successfully
SLO, objectivea target for an SLI99.9% of requests succeed, over 30 days
SLA, agreementa contract with consequences for missing the SLOs in itservice credits if the objective is missed

To tell an SLO from an SLA, ask what happens when it is missed. If nothing explicit happens, it is an SLO. Latency objectives are usually layered, as in the SRE book's example: 90% of calls under 1 ms, 99% under 10 ms and 99.9% under 100 ms. Do not simply adopt today's performance as the target, avoid absolutes such as "always available", and keep the internal target tighter than anything you publish.

Error budgets#

An SLO of 99.9% also says that 0.1% of requests may fail. That allowance is the error budget, and it turns "how reliable should we be?" into something a team can spend:

SLOBudget over 30 days, if everything fails
99%7.2 hours
99.9%43.2 minutes
99.95%21.6 minutes
99.99%4.32 minutes

While budget remains, new releases can go out. When it is used up, the SRE book's policy is that releases halt until reliability recovers. The nines calculator works through the same arithmetic.

Alerting#

Page people for symptoms, not causes. High CPU may mean nothing, while users getting errors always matters. The SRE book asks that every page be urgent, actionable and require human judgment. A page that only needs a robotic response should be automated.

The best symptom to alert on is the rate at which the budget is being spent. Burn rate measures that: at a burn rate of 1 the budget lasts exactly the window, and at 2 it lasts half that. For a 99.9% SLO over 30 days, a burn rate of 1 means a 0.1% error rate and a burn rate of 10 means 1%, which uses up the budget in 3 days. The SRE Workbook recommends alerting at several speeds:

SeverityBurn rateLong windowShort windowBudget spent when it fires
Page14.41 hour5 minutes2%
Page66 hours30 minutes5%
Ticket13 days6 hours10%

An alert fires only if both windows are over the burn rate. The long window means it is sustained, and the short one means it is still happening, so the alert clears soon after the errors stop.

Low traffic breaks this. With 10 requests an hour, one failure is a 10% error rate, which would page immediately against a 99.9% SLO. The Workbook's options include generating synthetic traffic, combining related services, and lowering the SLO or lengthening the window.

In an interview#

When you finish a design, a few sentences are usually enough:

  • Define healthy from the user's side. Name one or two SLIs and an SLO, such as the share of timeline reads that succeed within a latency target.
  • Say what you would graph: RED for each service, plus saturation for the part most likely to run out, such as the queue length or the database connections.
  • Say how you would debug. Propagate a trace ID through every service and put it in every log line.
  • Say what pages someone, and that it is a burn rate on the SLO rather than a CPU threshold.
  • Mention the cost. Sample traces, keep label values bounded, and limit log volume.

Sources and further reading#

Esc
↑ ↓ to move↵ to open