Case study: Ad click aggregation
Design a real-time ad click aggregation pipeline with event-time tumbling windows, watermarks for late clicks, and exactly-once billing.
35 min9 steps
By the end you can
- Estimate peak click ingestion rate, deduplication state memory, and write reduction from 1-minute window rollups
- Assign events to event-time tumbling windows and use watermarks and allowed lateness to handle delayed mobile clicks
- Prevent double-billing on client retry or stream worker crash using impression deduplication and idempotent window upserts
- Predict how the pipeline behaves when stream aggregators crash, a Super Bowl campaign spikes, or the OLAP store slows down
Separate user redirect latency from billing accuracy
Collect
- Signed clicks; immediate browser redirect; dedup invalid clicks
Guarantees
- No accepted click lost; exactly-once billing
Targets
- Redirect p99 <50 ms; dashboards 10-30 s fresh; minute windows
Estimate stream volume and rollup compression
Clicks
11.5k/s
1B/day / 86,400; peak 50k/s; 2M active ads
Ingress
15 MB/s
50k * 300 B; 16-32 log partitions
Dedup
4.3 GB
50k * 3,600 * 24 B; attribution state about 67 GB
Rollup writes
3.3k/s
200k clicked ads/minute; 15x write reduction
Design the collector and query APIs
Validate
GET /v1/click: HMAC impressionId, adId, price, issuedAt
Redirect
Durably append event -> 302 Location
Query
GET /v1/ads/{adId}/metrics?from=&to=&granularity=1m
Return
Watermark, clicks, spendCents; 5-second cache
Model idempotent window rollups
Window key
adId + windowStartMinute -> clicks, spendCents, updatedAt
Checkpoint
Processor persists running totals
Replay
PUT clicks=1420 again; never increment replayed deltas
Trace the streaming pipeline
Put the life of an ad click event in order from browser tap to advertiser dashboard.
- 1Stream aggregator consumes the event and checks the partitioned dedup store for the impressionId
- 2Click collector verifies the signed impression token, appends the event to the adId's stream partition, and returns 302
- 3When the watermark passes the window end, the aggregator upserts `(adId, windowStart, totalClicks, totalSpend)` to the OLAP store
- 4Aggregator adds the click to the in-memory 1-minute event-time window state for `(adId, windowStart)`
Deep dive: event time, watermarks, and hot campaigns
Event time
- 12:00:45 belongs in 12:00 window, even if received 12:03:10
Watermark
- Fire at 12:01; accept corrections until 12:11 with 10-minute lateness
Hot campaign
- Salt adId across 8 workers; merge 5-second partials
Compare the alternatives
ad-click-non-idempotent
- Replay increments twice; duplicateEffect
ad-click-sync
- Billing DB blocks redirects; goodput <85%
Hybrid reconciliation
- Streaming dashboards; raw archive -> daily authoritative invoice
Run the failure scenarios
Inject
Workers down 1.3-1.8 s
Observe
Replay after 400 ms
Recover
Dedup + PUT; no duplicateEffect; >99% goodput
Inject
Traffic x3 at 1.2-2.2 s
Observe
Log buffers burst
Recover
Redirects fast; consumer lag drains
Inject
OLAP latency x4
Observe
Rollups lag
Recover
Redirect isolated; cache shields queries
Wrap up and self-assess
Pace
Stop serving near budget exhaustion within seconds
Clock
Use impression server time + elapsed delta
Audit
Reconcile raw clicks and billing ledger
Answer this step to continue.