Case study: Metrics monitoring and alerting

What distinguishes a time-series metric from a structured log event?

At 5 million samples/s, how much daily storage does Gorilla compression (~1.37 B/sample) save compared to raw 16-byte `(timestamp, float64)` pairs?

Why does the query frontend snap `start` and `end` timestamps to multiples of `step` (e.g., 15 seconds) before checking the query cache?

A developer adds `user_id` (10 million distinct users) as a label on `http_requests_total`. What breaks first?

During an incident, engineers run expensive 30-day ad-hoc queries that overload the historical query tier. How do you ensure alerting rules still fire on time?

When downsampling 10-second samples into 1-hour rollup chunks, why should you store `(min, max, sum, count)` instead of just the 1-hour `average`?

An ingester holding 90 minutes of uncompressed/unflushed 2-hour head chunks in RAM crashes. How does the system avoid losing those 90 minutes of metrics?

Your company's Alertmanager routes notifications through the internal corporate chat server, which runs on the same Kubernetes cluster as production. Why is that dangerous?