Metrics monitoring and alerting (Prometheus / Datadog)
Collect tens of millions of time-series samples per second from hosts and services, compress them in memory, serve fast dashboard range queries, and evaluate alerting rules without failing when the fleet being monitored is itself on fire.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Distributor 1, Distributor 2
- Distributor 1 (service) · calls: Partitioned metric stream, Query & step cache, Time-series store (Gorilla chunks)
- Distributor 2 (service) · calls: Partitioned metric stream, Query & step cache, Time-series store (Gorilla chunks)
- Partitioned metric stream (queue) · calls: none
- TSDB ingesters (in-memory head) (worker) · calls: Time-series store (Gorilla chunks)
- Query & step cache (cache) · calls: none
- Time-series store (Gorilla chunks) (kvStore) · calls: none
Requirements
Functional
- Ingest time-series samples `(metricName, labels{}, timestampMs, float64Value)` from hundreds of thousands of targets.
- Query time-series over a time range with label selectors, aggregations (`sum`, `rate`, `histogram_quantile`), and downsampled resolutions.
- Continuously evaluate alerting rules against recent windows and fire deduplicated notifications with hysteresis (`for: 5m`).
Non-functional
- Write-heavy (~85–95% writes): sustain millions of samples/s with zero impact on monitored applications.
- Dashboard and alert evaluation queries return in <500 ms p99.
- High availability during incidents: the monitoring system must stay up when a zone or workload spikes.
Worked estimates
Active time series
≈ 50M active time series (cardinality is the primary memory driver)
200k hosts/pods × 250 series each
Sample ingestion rate
≈ 5M samples/s (~5,000 batched HTTP requests/s at 1,000 samples/batch)
50M series ÷ 10 s scrape/push interval
Raw vs Gorilla compressed size
5M/s × 1.37 B × 86,400 ≈ 590 GB/day (~12× smaller than raw)
16 B raw (8 B ts + 8 B float) → ~1.37 B/sample with delta-of-delta + XOR compression
In-memory head state
≈ 65 GB RAM across the ingester ring before flushing 2-hour blocks to object storage
50M active series × ~300 B metadata + 2 h head chunk
Downsampled retention
long-range 90-day queries read 6× fewer points than raw
raw 10s for 15d + 1m rollup (min/max/sum/count) for 90d + 1h for 2y
API
POST /v1/metrics/write
Request: Protobuf/Snappy batch of `{ labels[], samples[{ ts, value }] }`
Response: 200 · 429 series cardinality limit exceeded
Agents batch 500–2,000 samples per request.
GET /v1/query_range?query=...&start=...&end=...&step=15s
Request: PromQL-style selector + step
Response: 200 { resultType: 'matrix', result[{ metric, values[] }] }
POST /v1/rules
Request: { expr, forDuration, severity, labels }
Response: 201 { ruleId }
Evaluated every 15–30 s by the rule engine against the query tier.
Data model
- series — store: tsdb
Each unique label set (`__name__="http_requests_total", service="checkout", status="500"`) is hashed to a 64-bit `seriesId`. High-cardinality labels like `userId` or `requestId` create millions of new `seriesId`s (cardinality explosion) and must be rejected or stripped.
Ingesters append samples for each `seriesId` to an in-memory 2-hour head chunk (Gorilla compressed) backed by a disk WAL, and flush immutable 2-hour blocks + inverted label index to object storage.
A step-aligned query frontend splits `query_range` into day/hour sub-queries, caches immutable historical intervals, and merges them with the live head chunk.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Gorilla time-series compression (delta-of-delta + XOR)
Facebook's Gorilla paper showed that within a single time series, timestamps arrive at nearly constant intervals (e.g., every 10,000 ms) and float values change slowly. Storing the *delta-of-delta* `(t_n - t_{n-1}) - (t_{n-1} - t_{n-2})` means 96% of timestamps equal 0 and take a single `0` bit! Float64 values are XORed with the previous value: if identical, store one `0` bit; otherwise store the leading/trailing zero count and the changed middle bits. Average size drops from 16 bytes to ~1.37 bytes per sample.
The enemy is cardinality, not sample rate
Adding 10× more samples to an existing series only appends 1.37 bytes to an existing chunk. Adding a label like `user_id` or `url_path` creates millions of *new* series, each requiring an in-memory head chunk, inverted-index postings list entries (`label=value → [seriesId...]`), and compaction work. Production systems enforce per-tenant active-series quotas at the distributor and drop or relabel unbounded tags before they reach the ingesters.
Reliable alerting: hysteresis, deduplication, and dead-man's switch
Alert rules run every 15–30 s against recent in-memory head chunks (never cold disk). To avoid paging on transient 10-second blips, rules require a `for: 5m` pending duration (hysteresis) before transitioning `INACTIVE → PENDING → FIRING`. Multiple HA alert evaluators send firing alerts to an Alertmanager cluster that deduplicates by alert fingerprint, groups related host alerts into one notification, and sends a continuous heartbeat ('dead-man's switch') to an external pager so you know if the alerting pipeline itself dies.
Failure scenarios
Major incident: metric volume and dashboard queries double
Scenario and expected result
During an outage from t = 1.2 s to 2.2 s, autoscaled pods emit 2× more metrics and engineers refresh dashboards at 2× the normal rate.
Expected: The partitioned write stream buffers the ingestion burst while step-aligned query caching shields the TSDB from repeated dashboard refreshes.
TSDB ingesters restart during a rolling deploy
Scenario and expected result
The ingester worker pool restarts from t = 1.3 s to 1.8 s.
Expected: Distributors continue accepting metric batches into the durable partitioned stream without dropping samples; ingesters drain the backlog as soon as they come back.
The query cache restarts cold
Scenario and expected result
At t = 1.5 s the query cache loses its state.
Expected: Queries fall through to the in-memory/chunk TSDB store and repopulate the cache within milliseconds; ingestion is completely unaffected.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
Synchronous uncached range scans without stream buffering
- Storing each `(seriesId, timestamp, value)` as an individual row without an in-memory head chunk or query cache turns range queries and writes into full scans. In the simulator, routing writes and uncached scans straight to the store pushes p99 latency above 18 ms (more than 4× the buffered chunk design).
Reference
- Inspect the architecture above and run the same scenario.
Pull (Prometheus scrape) vs Push (agents / OpenTelemetry)
- Pull lets the monitoring server control scrape rate, discover targets via service discovery, and tell immediately when a target is down (`up == 0`). Push works across NAT/firewalls, supports short-lived serverless jobs, and scales centrally via load-balanced distributors (Cortex / Mimir / VictoriaMetrics combine Prometheus pull at the edge with remote-write push to central storage).
Reference
- Inspect the architecture above and run the same scenario.
No query step alignment or caching
- When an incident hits, 200 engineers open the same Grafana dashboard and refresh it every 10 seconds with slightly different `now` timestamps (`12:00:01`, `12:00:03`). Aligning `start`/`end` to the `step` grid (`15s`) and caching historical sub-intervals turns 200 identical TSDB scans into 1.
Clarifying questions
- How many hosts/containers and how many active time series do we expect, and at what collection interval?
- Are we designing for pull (Prometheus-style scraping), push (agents/remote-write), or both?
- What retention resolution is needed (e.g., 10s raw for 15 days, 1m rollups for 90 days, 1h for 2 years)?
- How quickly must alerts fire, and how should the alerting pipeline itself be monitored?
Common mistakes
- Storing every metric sample as an individual row in a B-tree SQL database.
- Allowing unbounded labels (`userId`, `requestId`, raw URLs) into metric tags, causing a cardinality OOM.
- Sending every sample as a separate HTTP request instead of batching hundreds of samples per payload.
- Running alert evaluations against slow cold storage instead of the in-memory recent head chunks.
- Hosting the monitoring and alerting system inside the exact same cluster and dependencies it monitors.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Collect and graph metrics.
- 2/4 Adds alerting.
- 3/4 Quantifies write-heavy ratio, query latency, retention tiers, and HA during incidents.
- 4/4 Distinguishes metrics (aggregated counters/gauges/histograms) from high-cardinality logs and traces.
estimates / levels 1-4
- 1/4 None.
- 2/4 Samples/s only.
- 3/4 Calculates active series (50M), samples/s (5M/s), and Gorilla ~1.37 B/sample compression.
- 4/4 Sizes in-memory head chunk RAM and downsampled retention tiers.
api / levels 1-4
- 1/4 Write and read endpoints.
- 2/4 Batched writes and range query.
- 3/4 Step-aligned range queries, label selectors, and 429 on cardinality quota.
- 4/4 Explains client-side batching and scrape vs remote-write protocols.
data model / levels 1-4
- 1/4 Metrics table.
- 2/4 Series + samples.
- 3/4 Series ID + inverted label index + compressed time-ordered chunks (head + 2h blocks).
- 4/4 Explains cardinality explosion and downsampling rollups (`min/max/sum/count`).
high level / levels 1-4
- 1/4 Agent → DB.
- 2/4 Collectors + TSDB + dashboards.
- 3/4 Distributors, partitioned WAL/stream, in-memory head ingesters, object store blocks, query cache.
- 4/4 Includes decoupled alert rule evaluators + HA Alertmanager with deduplication.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions TSDB.
- 3/4 Explains Gorilla delta-of-delta + XOR compression and inverted label postings.
- 4/4 Details cardinality limits, query splitting/step alignment, and dead-man's switch alerting.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions dashboards.
- 3/4 Names cardinality spikes and incident query storms as primary failure modes.
- 4/4 Proposes per-tenant shuffle sharding and meta-monitoring in a separate failure domain.