Stream processing and windows
Assign events to windows, reason about watermarks and late updates, and size bounded deduplication state without claiming global exactly-once effects.
26 min8 steps
By the end you can
- Assign event timestamps to aligned half-open tumbling windows
- Calculate state cleanup thresholds and choose a policy for late events
- Estimate deduplication memory from unique arrival rate and retention
- Choose keys and sink semantics that keep replay and late corrections safe
An event has more than one clock
Event time: when it happened. Processing time: when the pipeline sees it. Validate timestamps at ingestion; a far-future clock distorts progress.
12:01
Purchase happens (event time)
12:01–12:08
Phone offline
12:08
Pipeline receives it (processing time)
A window boundary needs a convention
Tumbling window, [start, end): start = floor(timestamp / width) × width. State the alignment; local days are not always 24 hours.
t = 125
[120, 180)
floor(125/60) × 60 = 120
t = 180
[180, 240)
end is excluded above
Loading question…
Choose the grouping the question asks for
Pick the window for the question, not for a convenient timer.
Tumbling
- Fixed, disjoint
- Per-minute totals
Sliding
- Overlaps
- Never sum outputs as unique events
Session
- Joined by inactivity gap
- Late data can merge sessions
A watermark estimates progress
A watermark estimates event-time progress. It is not a barrier: older events can still arrive. The slowest input (or an idle partition) holds it back. Advance fast for latency, wait for completeness.
12:05
Watermark: progress estimate
12:06
Window emitted
12:09
Event stamped 12:03 arrives: late
Late events need a visible result policy
Window ends at 120 s, allowed lateness 30 s: keep state until the watermark reaches 150 s. Early results are provisional; updates replace, never add. Count and report dropped late data.
120 s
Window closes; first output
120–150 s
Late events revise the output
150 s
State cleaned; later events dropped or side-output
Loading question…
Deduplication spends bounded state
Bounded state: IDs per second × retention × bytes per entry. Use a stable event ID. A duplicate after expiry is accepted again.
IDs held
300,000
State
19.2 MB
before replicas and checkpoints
Loading question…
Replay safety must reach the sink
Replay can repeat external effects. Make the sink idempotent: upsert by metric, entity and window with a version check, or send uniquely identified deltas.
Check your mastery
Score 80% or more. State the timestamp convention, progress measure and retained horizon.
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
Design a revenue counter: event ID, partition key, window, provisional-output rule, late-data path and sink recovery. These questions model event-time arithmetic, not a full streaming runner.
Answer this step to continue.