Ad click event aggregation
Ingest tens of thousands of ad clicks per second, deduplicate retries and fraudulent double-clicks, and produce per-minute billing and dashboard counts within seconds. Because clicks are money, the pipeline needs event-time windowing, watermarks for late mobile events, and end-to-end exactly-once accounting.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Click collector 1, Click collector 2
- Click collector 1 (service) · calls: Click event log (by adId), Dashboard cache, Minute rollups & OLAP store
- Click collector 2 (service) · calls: Click event log (by adId), Dashboard cache, Minute rollups & OLAP store
- Click event log (by adId) (queue) · calls: none
- Stream aggregators (worker) · calls: Click-ID dedup state, Minute rollups & OLAP store
- Click-ID dedup state (dedupStore) · calls: none
- Dashboard cache (cache) · calls: none
- Minute rollups & OLAP store (kvStore) · calls: none
Requirements
Functional
- Record every ad click event (`clickId`, `impressionId`, `adId`, `campaignId`, `eventTime`, user context) and redirect the user to the landing page with minimal delay.
- Aggregate click counts and spend per `adId` and `campaignId` in 1-minute tumbling windows for live advertiser dashboards and budget pacing.
- Deduplicate repeated clicks carrying the same `clickId` / `impressionId` and filter invalid or unserved impressions.
Non-functional
- Click redirect latency under ~50 ms p99; dashboard freshness within ~10–30 seconds.
- No double-billing when clients retry or stream workers crash and replay a partition.
- Zero lost clicks once accepted by the collector (durable partitioned log + raw archive for offline reconciliation).
Worked estimates
Click ingestion
≈ 11.5k clicks/s average, ≈ 50k/s peak during major events
1B clicks/day ÷ 86,400
Impressions (join state)
≈ 4.2B impressions/hour ≈ 67 GB across stream partitions
100B impressions/day (~1% CTR) × 16 B key × 1 h attribution window
Raw log bandwidth
≈ 15 MB/s ingress to Kafka; ≈ 300 GB/day compressed in object storage
50k clicks/s × 300 B event payload
Minute rollup writes
pre-aggregating in 1-minute windows cuts OLAP store writes from 50k/s to ~3.3k/s
2M active ads × 1 row/minute (sparse: ~200k ads clicked in any given minute)
Dedup state
≈ 4.3 GB across the stream processor state stores
50k clicks/s × 3,600 s TTL × 24 B
API
GET /v1/click?clickId=...&adId=...&dest=...
Request: Signed impression token + client timestamp
Response: 302 Found → advertiser landing URL
Appends to the durable click stream before or alongside returning 302.
GET /v1/ads/{adId}/metrics?from=...&to=...&granularity=1m
Request: Time range and breakdown dimensions
Response: 200 { series[{ windowStart, clicks, spendCents }], watermark }
Data model
- aggregates — store: olapdb
- billingLedger — store: olapdb
aggregates is keyed by `(adId, windowStartMinute)` in a real-time OLAP store (ClickHouse / Druid / Pinot) or wide-column store; stream workers write deterministic window totals so replays overwrite rather than double-count.
billingLedger records deduplicated billable click events (also dumped to Parquet in object storage) for daily batch reconciliation against the real-time stream.
The stream processor maintains keyed rocksdb state partitioned by `adId` (or `impressionId`) for deduplication and 1-minute window accumulators.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Event time, watermarks, and allowed lateness
Mobile clients buffer ad clicks while offline in a subway and upload them minutes later. Aggregating by *processing time* (when the server receives the click) spikes the wrong minute bucket. Instead, bucket by *event time* (captured when the ad impression/click occurred, validated against server impression timestamps to prevent spoofing). The stream processor tracks a *watermark* `W` — a monotonic lower bound on event time — and emits the `[12:00, 12:01)` window once `W >= 12:01`. Late events arriving within an *allowed lateness* horizon (say, 10 minutes) update the emitted bucket; ultra-late events go to a side output for daily billing reconciliation.
End-to-end exactly-once accounting
At-least-once ingestion plus retries means duplicates enter the pipeline at two places: mobile client retries and stream worker crashes. First, when the ad server renders an ad, it issues a signed `impressionId`; the collector and stream processor deduplicate on `impressionId` in a partitioned state store (RocksDB backed by checkpointed changelog topics). Second, when a 1-minute window closes, the processor writes the full window total `(adId, windowStart) → totalClicks` (an idempotent `PUT` keyed by window ID) or commits Kafka offsets and sink transactions together, rather than issuing blind `+1` increments to the external database.
Handling hot ads and budget pacing
Partitioning the stream by `adId` keeps each ad's window state on a single worker, except when one viral campaign receives 20% of all clicks. Solve hot keys with a *two-stage local-then-global aggregation*: collectors or stage-1 workers append a random shard suffix `adId#0..7` and emit partial 5-second sub-aggregates, reducing message volume by 1000× before stage-2 workers merge the 8 partials per `adId` and check remaining campaign budget.
Failure scenarios
Stream aggregator workers crash mid-window
Scenario and expected result
All stream aggregator workers crash from t = 1.3 s to 1.8 s. Unacknowledged click events in the stream reappear after the 400 ms visibility timeout and are reprocessed.
Expected: Click redirects keep succeeding at full speed because collectors only publish to the partitioned log. When the workers recover and replay the redelivered clicks, the dedup store and idempotent window writes prevent any duplicate billing.
Super Bowl ad airs: click rate triples
Scenario and expected result
From t = 1.2 s to 2.2 s traffic triples to 9,000 rps as a broadcast ad drives a sudden burst of clicks.
Expected: The partitioned click log buffers the burst so click redirects stay fast; consumer lag rises for a moment and drains once the spike passes.
The OLAP rollup store slows down 4×
Scenario and expected result
Segment merges slow the OLAP store by 4× from t = 1.4 s to 2.2 s.
Expected: Click redirects are completely isolated by the stream log, and dashboard reads are shielded by the 5-second dashboard cache.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
Non-idempotent counter increments on worker replay
- If stream workers simply increment a database counter (`clicks = clicks + 1`) without deduplicating by `clickId` or writing deterministic window snapshots, any worker crash that replays unacknowledged messages bills the advertiser twice. Run the worker-crash scenario against this design to observe `duplicateEffect` violations.
Reference
- Inspect the architecture above and run the same scenario.
Synchronous database writes and rollup scans in the click redirect path
- Writing and scanning rollups synchronously in the OLAP store before redirecting removes the stream queue, but couples user click latency and availability to the database. Under load, redirect p99 latency more than triples compared to asynchronous stream ingestion.
Reference
- Inspect the architecture above and run the same scenario.
Pure batch processing vs hybrid reconciliation
- Hourly Spark/MapReduce jobs over object storage give simple, exact numbers with zero late-event headaches, but advertisers cannot see live campaign performance or stop an overspending budget for an hour. Production ad systems run a fast streaming path for live dashboards and pacing, plus a daily batch reconciliation job for final invoice adjustments.
Clarifying questions
- Are we only counting clicks, or joining clicks against ad impressions to verify attribution and compute CTR?
- How fresh do advertiser dashboards and budget-pacing decisions need to be (seconds vs minutes)?
- How late can offline mobile click events arrive, and how should late events affect already-closed minute buckets?
- What is the peak click rate, and can a single viral campaign dominate traffic?
Common mistakes
- Writing directly to a database in the synchronous click-redirect path.
- Incrementing an external counter (`clicks = clicks + 1`) from an at-least-once stream worker without idempotency.
- Windowing by server processing time instead of event time, corrupting minute buckets when queues lag or mobile devices reconnect.
- Trusting client-generated `clickId`s without verifying a server-signed `impressionId`.
- Keeping unbounded deduplication state in memory without a watermark-driven TTL.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Count clicks per ad.
- 2/4 Adds redirect latency and dashboard queries.
- 3/4 Specifies 1-minute event-time windows, deduplication, and no double-billing.
- 4/4 Covers late mobile events, impression-to-click attribution, and budget pacing.
estimates / levels 1-4
- 1/4 None.
- 2/4 Click QPS only.
- 3/4 Computes peak click QPS (~50k/s), dedup state memory, and rollup write reduction.
- 4/4 Sizes the impression-join state (~67 GB/h) and raw archive bandwidth.
api / levels 1-4
- 1/4 Click and query endpoints.
- 2/4 302 redirect on click.
- 3/4 Signed impressionId/clickId token and time-range rollup query with watermark.
- 4/4 Explains why `impressionId` must be server-signed to prevent fabricated clicks.
data model / levels 1-4
- 1/4 Clicks table.
- 2/4 Clicks + aggregates table.
- 3/4 Raw immutable event log (S3/Parquet) + `(adId, windowStart)` OLAP rollups + dedup state.
- 4/4 Idempotent window upsert schema that tolerates stream replay and late updates.
high level / levels 1-4
- 1/4 Collector writes DB.
- 2/4 Collector → Kafka → workers → DB.
- 3/4 Partitioned log, stateful stream processor with dedup/windows, OLAP store, and dashboard cache.
- 4/4 Adds raw archival sink and batch reconciliation path for invoice accuracy.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions Kafka and Flink.
- 3/4 Explains event-time tumbling windows, watermarks, and allowed lateness.
- 4/4 Details end-to-end idempotency on crash replay and two-stage combiner for hot ads.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Names watermark lag, hot ad partitions, and click fraud as key risks.
- 4/4 Sets SLOs on redirect latency and watermark lag, plus daily stream-vs-ledger audits.