What You'll Master Here
Stream processing answers questions over data that never ends, so you slice it into windows, reason in event time, and keep state, instead of waiting for "all the data".
Chapter 13 gave you the streaming architecture: a partitioned log read by consumers. This chapter is about what those consumers actually compute. Stream processing is the art of producing answers from data that never stops arriving, and it forces you to rethink ideas that were trivial in batch.
In batch, "count the events" is one line, because you have all the events. In a stream there is no "all"; more is always coming. So stream processing introduces windows (slice the endless stream into finite buckets), event time (reason about when things happened, not when they arrived), and state (remember things across events).
By the end you will understand stateless versus stateful processing, the three window types and when to use each, why event time and watermarks are the heart of correct streaming, and what Flink, Spark Structured Streaming, and Kafka Streams give you. These are the concepts behind every real-time aggregate you have ever seen.
Stream processing answers questions over data that never ends, so you slice it into windows, reason in event time, and keep state, instead of waiting for "all the data".
Real-time metrics, fraud scores, and live dashboards are all stream processing. The subtle parts, windows, event time, watermarks, are exactly where naive streaming silently produces wrong numbers, so mastering them is what makes real-time data trustworthy.
- stream processing
- Computing results continuously over an unbounded flow of events.
- window
- A finite slice of an endless stream over which you aggregate.
- state
- Information a processor remembers across events (running counts, last value).
- event time
- When an event actually happened, as opposed to when it was processed.
Applying batch instincts ("count all the rows") to an endless stream. There is no "all"; without windows and event time, your counts are arbitrary and often wrong.
Aggregate streams over windows, never "everything".
Reason in event time, not arrival time.
Treat state as a first-class, managed concern.
Batch waits for all the data; streaming never can. So you replace "all the data" with windows, replace wall-clock with event time, and carry state forward. Those three shifts are the whole chapter.
Read stateless vs stateful first, then windowing, then event time and watermarks (the hardest and most important), then the tools.
Stream processing computes over endless data using windows, event time, and state; those three ideas replace the batch luxury of having "all the data".
- Explain why "count all the events" does not work on a stream.
- Define event time versus processing time.