Metrics monitoring and alerting (Prometheus / Datadog)

Collect tens of millions of time-series samples per second from hosts and services, compress them in memory, serve fast dashboard range queries, and evaluate alerting rules without failing when the fleet being monitored is itself on fire.

Notes and ratings save in this browser. Completing self-assessment records your progress.

Duration 45:00

Requirements checklist

Ingest time-series samples `(metricName, labels{}, timestampMs, float64Value)` from hundreds of thousands of targets.Query time-series over a time range with label selectors, aggregations (`sum`, `rate`, `histogram_quantile`), and downsampled resolutions.Continuously evaluate alerting rules against recent windows and fire deduplicated notifications with hysteresis (`for: 5m`).Write-heavy (~85–95% writes): sustain millions of samples/s with zero impact on monitored applications.Dashboard and alert evaluation queries return in <500 ms p99.High availability during incidents: the monitoring system must stay up when a zone or workload spikes.

Clarifying questions

How many hosts/containers and how many active time series do we expect, and at what collection interval?
Are we designing for pull (Prometheus-style scraping), push (agents/remote-write), or both?
What retention resolution is needed (e.g., 10s raw for 15 days, 1m rollups for 90 days, 1h for 2 years)?
How quickly must alerts fire, and how should the alerting pipeline itself be monitored?

Open a practice canvas (new tab)Estimation calculators (new tab)

requirements

Suggested time: about 6 minutes. Distinguishes metrics (aggregated counters/gauges/histograms) from high-cardinality logs and traces.

Notes for requirements