Metrics, logs and traces
Pick the right signal for an operational question, alert on symptoms, keep metric cardinality bounded, and budget what traces cost to keep.
25 min7 steps
By the end you can
- Choose between metrics, logs and traces for a given operational question
- Alert on user-visible symptoms using the four golden signals rather than on causes
- Predict how labels multiply metric time series and keep cardinality bounded
- Explain how trace context propagates across services and why the sampling decision must travel with it
- Estimate stored trace volume from traffic, sampling rate and span size
Three signals, three different questions
Metrics
- Is it real, how bad?
- Cheap aggregates
- Drive alerts
Logs
- What happened to this request?
- Structured, with trace ID
- Cost per event
Traces
- Where did the time go?
- Tree of spans
- Sampled (Dapper)
Page on symptoms, investigate causes
Golden
L T E S
latency, traffic, errors, saturation
RED (services)
rate, errors, duration
USE (resources)
utilisation, saturation, errors
Page on
symptoms
causes go on dashboards
Labels multiply
20 endpoints x 5 status x 100 pods
10,000
per bucket
Add user_id, 1M users
10 billion
backend falls over
Loading question…
Which labels are safe?
Safe
- Route template /users/:id
- Status class 5xx
- Region, method
Unsafe
- Raw path
- user_id, request ID
- Error message text
Follow the trace context
Put the steps of tracing one request through a gateway and an orders service in order.
- 1The gateway calls the orders service with traceparent carrying the trace ID, its span ID and the sampled flag
- 2Each service ends its spans and exports them; the backend joins them by trace ID into one tree
- 3The orders service queries the database inside its own child span
- 4The orders service starts a child span whose parent is the gateway's span ID
- 5The gateway receives a request with no traceparent, creates a trace ID and a root span, and makes the sampling decision
Sample, and budget what you keep
Head-based
- Decide at the root
- Cheap, consistent
- Keeps 1% of errors too
Tail-based
- Buffer, then keep errors and slow
- Needs a collector tier
Loading question…
Check your mastery
Question
which hop holds the time?
Score 80% or more. Open the cache-aside design and send one request from the Trace tab. Next: SLIs, SLOs and error budgets.
Mastery check
4 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
Answer this step to continue.