What You'll Master Here
Event time = when it happened. Processing time = when Spark saw it. Windows group by event time; the watermark is the cutoff that says "stop waiting for stragglers older than this" so state can be released.
The hard part of streaming is time. Events happen at one moment (event time) but arrive at another (processing time), often late, out of order, or delayed by network hiccups. To compute correct windowed results, "how many orders per 5-minute window?", you must reason about event time, not arrival time. This chapter is about doing that correctly.
You will learn event vs processing time, time windows (tumbling and sliding), watermarks (how Spark decides when a window is "done" and how long to wait for late data), stateful operations (which hold information across batches), and how checkpointing delivers exactly-once. This is the most conceptually demanding streaming material, and the most valuable.
With diagrams of tumbling vs sliding windows and the watermark mechanism, plus a windowed-aggregation example. Master this and you can build correct real-time aggregations.
Event time = when it happened. Processing time = when Spark saw it. Windows group by event time; the watermark is the cutoff that says "stop waiting for stragglers older than this" so state can be released.
Late and out-of-order data is the reality of streaming. Watermarks and event-time windows are how you get correct results without keeping state forever. This is exactly the depth that distinguishes a real streaming engineer in interviews and production.
- event time
- The timestamp when an event actually occurred (carried in the data).
- processing time
- When Spark processes the event, which can lag event time.
- window
- A bucket of event time (e.g., each 5-minute interval) to aggregate over.
- watermark
- A moving threshold of event time past which late data is dropped and state freed.
Aggregating by processing time when the question is about event time. Late/out-of-order events land in the wrong window, giving wrong counts, use event time + watermarks.
Aggregate by event time for correctness, not arrival time.
Set a watermark on event-time aggregations to bound state.
Choose tumbling vs sliding windows based on whether you need overlap.
Correct streaming aggregation uses event time and a watermark: group events into event-time windows, and use the watermark to decide how long to wait for late data before finalising a window and discarding its state.
Streaming correctness hinges on event time, not arrival time. Windows group events by event time; watermarks bound how long to wait for late data and let Spark release state. Checkpointing makes the whole thing exactly-once.
- Why aggregate by event time instead of processing time?
- What problem does a watermark solve?