What You'll Master Here
A stream is a table you never stop appending to. Your query is a standing question Spark keeps re-answering on the new rows. Same DataFrame API, incrementally executed.
Structured Streaming is Spark's engine for processing continuous data, and its defining idea is beautifully simple: treat a stream as an unbounded table that grows over time. You write almost the same DataFrame code as for batch; Spark incrementally runs it as new data arrives. The same Catalyst, the same operators, the same mental model, applied to never-ending input.
This chapter covers that core abstraction, the streaming sources and sinks, output modes (append/update/complete), triggers (how often to process), and the micro-batch vs continuous execution models. It also introduces checkpointing for streaming, which gives fault tolerance and exactly-once semantics.
With a diagram of the unbounded-table model and a worked readStream → transform → writeStream pipeline. The headline: if you know batch DataFrames, you already mostly know streaming.
A stream is a table you never stop appending to. Your query is a standing question Spark keeps re-answering on the new rows. Same DataFrame API, incrementally executed.
Real-time pipelines (fraud detection, monitoring, live dashboards) run on streaming. Structured Streaming's "batch code on an unbounded table" model is why Spark engineers can build them without learning a separate paradigm, a key, marketable skill.
- unbounded table
- The model: a stream is a table that grows as data arrives.
- readStream / writeStream
- The streaming counterparts of read/write for sources and sinks.
- output mode
- What to emit each batch: append, update, or complete.
- trigger
- How often the streaming query processes new data.
Assuming streaming needs a completely different API from batch. You over-complicate it, the DataFrame transformations are almost identical; only the read/write and a few stream concepts differ.
Think of the stream as an ever-growing table and reuse batch logic.
Always configure a checkpoint location for fault tolerance.
Choose output mode and trigger deliberately for your use case.
Structured Streaming = run your batch DataFrame query continuously on an unbounded, growing table. Spark handles the incremental execution, state, and fault tolerance; you mostly write familiar transformations.
Structured Streaming models a stream as an unbounded table and runs your (mostly batch-identical) DataFrame query incrementally as data arrives. Output modes, triggers, and checkpointing are the streaming-specific pieces to learn.
- What is the core abstraction of Structured Streaming?
- How similar is streaming code to batch code, and what differs?