Windowing, watermarks, event-time vs processing-time, and stateful operators in Flink, Spark, and Kafka Streams.
⏱ 24 min readTopics chapter readerLevel · Streaming & Real-Time
01 · Orientation
What You'll Master Here
Stream processing answers questions over data that never ends, so you slice it into windows, reason in event time, and keep state, instead of waiting for "all the data".
⏱ 5 min · Topic 1 of 9
Chapter 13 gave you the streaming architecture: a partitioned log read by consumers. This chapter is about what those consumers actually compute. Stream processing is the art of producing answers from data that never stops arriving, and it forces you to rethink ideas that were trivial in batch.
In batch, "count the events" is one line, because you have all the events. In a stream there is no "all"; more is always coming. So stream processing introduces windows (slice the endless stream into finite buckets), event time (reason about when things happened, not when they arrived), and state (remember things across events).
By the end you will understand stateless versus stateful processing, the three window types and when to use each, why event time and watermarks are the heart of correct streaming, and what Flink, Spark Structured Streaming, and Kafka Streams give you. These are the concepts behind every real-time aggregate you have ever seen.
Core mental model
Stream processing answers questions over data that never ends, so you slice it into windows, reason in event time, and keep state, instead of waiting for "all the data".
Why it matters
Real-time metrics, fraud scores, and live dashboards are all stream processing. The subtle parts, windows, event time, watermarks, are exactly where naive streaming silently produces wrong numbers, so mastering them is what makes real-time data trustworthy.
stream processing
Computing results continuously over an unbounded flow of events.
window
A finite slice of an endless stream over which you aggregate.
state
Information a processor remembers across events (running counts, last value).
event time
When an event actually happened, as opposed to when it was processed.
A payments stream that never endsFour ordinary questions. Which of them can this stream answer without you inventing something?predict first
Common mistake
Applying batch instincts ("count all the rows") to an endless stream. There is no "all"; without windows and event time, your counts are arbitrary and often wrong.
Better habit
Aggregate streams over windows, never "everything".
Reason in event time, not arrival time.
Treat state as a first-class, managed concern.
The big idea
Batch waits for all the data; streaming never can. So you replace "all the data" with windows, replace wall-clock with event time, and carry state forward. Those three shifts are the whole chapter.
How to study this chapter
Read stateless vs stateful first, then windowing, then event time and watermarks (the hardest and most important), then the tools.
Remember this
Stream processing computes over endless data using windows, event time, and state; those three ideas replace the batch luxury of having "all the data".
Practice2 prompts
Explain why "count all the events" does not work on a stream.
Define event time versus processing time.
02 · The framework
Stateless vs Stateful Processing
each event alone, no memory, easy. Stateful: needs memory of the past, powerful but must be checkpointed, keyed, and bounded.
⏱ 6 min · Topic 2 of 9
Stream operations come in two kinds. Stateless operations treat each event in isolation: filter out bots, reshape a field, route by type. They need no memory, so they are simple, scale trivially, and recover instantly, each event is handled the same way regardless of what came before.
Stateful operations need memory of past events: a running count per user, a 5-minute average, "has this card been used in two countries in the last minute?". The processor must hold state and update it as events flow. This is where streaming gets powerful and hard, because that state must survive crashes, scale across partitions, and not grow without bound.
The practical lesson is to know which kind you are doing, and to push as much logic as possible into stateless operations, because state is the expensive part. When you do need state, you accept the machinery that comes with it: checkpointing it for recovery, keying it by partition, and bounding it (usually with windows) so it does not grow forever.
Core mental model
Stateless: each event alone, no memory, easy. Stateful: needs memory of the past, powerful but must be checkpointed, keyed, and bounded.
Why it matters
State is the source of most streaming complexity and most streaming bugs. Distinguishing stateless from stateful, and minimising state, is the first design decision that keeps a streaming job both correct and affordable.
stateless operation
Processes each event independently (filter, map, route); no memory needed.
stateful operation
Needs memory of past events (counts, averages, joins); requires managed state.
keyed state
State partitioned by a key (per user, per device) so it scales across partitions.
checkpoint
A saved copy of state so a crashed processor can recover without reprocessing everything.
Four streaming jobs · what each one has to rememberStateless costs nothing and is rarely the job. Keyed state is where the operational bill lives.no state
The job
How long state is kept
what it remembers
value
keys held
none
bytes per key
—
state size
0
what a restart costs
nothing — resume anywhere
Stateless: no memory, no checkpoint, no restoreA stateless job can be restarted, rescaled and replayed freely, because there is nothing to lose. It is also the shape almost nothing interesting has: as soon as the question involves “how many”, “since when” or “which other record”, you are in the column to the right.
Common mistake
Holding unbounded state (e.g. a count per user, forever). State grows without limit until the processor runs out of memory and fails.
Better habit
Push logic into stateless operations where possible.
Key and bound any state you must keep (usually with windows or TTLs).
Checkpoint state so processors recover after crashes.
Interview note
Identifying which parts of a streaming job are stateful, and how that state is keyed, bounded, and checkpointed, is a strong signal. State is where interviewers probe for real streaming experience.
Remember this
Stateless operations handle each event alone and are easy; stateful operations need keyed, bounded, checkpointed memory, which is where streaming’s power and difficulty both come from.
Practice2 prompts
Classify as stateless or stateful: drop bots, 5-minute average, reshape JSON, per-user running total.
Explain why unbounded state eventually crashes a streaming job.
03 · Before the definitions
Watch Six Payments Go Through
An event has two clocks — when it happened, and when you found out — and every idea in this chapter exists because those two clocks disagree.
⏱ 5 min · Topic 3 of 9
The next two topics use words like tumbling, sliding, session and watermark. This one uses none of them — it just shows six payments arriving and asks you to read the numbers.
One of the six is late: it happened at 12:00:30 and did not reach us until 12:02:30. Everything that makes streaming hard follows from that one event.
Fixed buckets — each event appears exactly once down the whole column.
Overlapping buckets — p2, p3 and p5 each appear twice, because two windows genuinely cover them.
Bounded by silence — a 65-second gap between p3 and p4 splits the events into two windows.
The late one — p5 is either in time, or corrects a number already published, or is thrown away, depending on two dials you control separately.
Core mental model
An event has two clocks — when it happened, and when you found out — and every idea in this chapter exists because those two clocks disagree.
Why it matters
These ideas are simple to watch and very hard to read. Seeing the counts change first means the definitions that follow are naming something you have already seen.
when it happened vs when you found out
Every event carries the time it occurred, and separately reaches you at some later time. p5 has 12:00:30 and 12:02:30. Almost every streaming decision is about which of those two you use.
how far along we believe time has got
The newest event time you have seen, minus however far you choose to hold the clock back. Once that number passes the end of a window, the window reports.
and whether it can still change afterwards
A separate choice from the one above. After a window reports, it is either finished for good, or stays open a while longer and re-reports a corrected number when something turns up late. Two dials, two different costs.
lag
How many events are waiting because the consumer is slower than the producer. It is a count, it is always visible, and it is the first number to put an alert on.
Six payments · 12:00 to 12:03The same six events, read four different ways. Watch which window each one lands in.tumbling
The events. One of them is late.
event
happened at
reached us at
how late
p1
12:00:10
12:00:11
1s
p2
12:00:55
12:00:56
1s
p3
12:01:05
12:01:06
1s
p4
12:02:10
12:02:11
1s
p5
12:00:30
12:02:30
2 min — late
p6
12:02:40
12:02:41
1s
p5 happened at 12:00:30, in the first minute — but it did not reach us until 12:02:30, by which time we had already seen events from 12:02. Everything below is about what to do with it.
One minute per bucket. Every event is in exactly one.
window
events in it
count
12:00
p1, p2, p5
3
12:01
p3
1
12:02
p4, p6
2
Count the ids: each one appears exactly once down the whole column. That is what makes these buckets “non-overlapping” — and note p5 is in the 12:00 bucket, where it happened, not the 12:02 bucket where it arrived.
10,000 payments a second arriving · one consumer handling 4,000The producer is faster than the consumer. Nothing errors. What happens over the next minute?lag growing
arriving 10,000/sprocessing 4,000/s
after
events waiting (lag)
10s
60,000
20s
120,000
30s
180,000
40s
240,000
50s
300,000
60s
360,000
A minute in, 360,000 events are queued — and the number only goes upThe gap is 6,000 events every second, and it never closes on its own.
Notice what did not happen: nothing crashed and nothing was lost. The broker held the backlog, which is exactly what a log is for. What you lose is freshness — the dashboard is reading a minute-old world and getting older.
Common mistake
Reading the overlapping-bucket table and concluding events were duplicated. You "fix" a correct pipeline by deduplicating, and the moving average you wanted becomes a set of disjoint buckets instead.
Treating "how long we wait" and "how long a number can still change" as one setting. They are two dials with different costs — one delays every answer, the other lets published answers move. Merging them is the most common way streaming lateness is misunderstood.
Assuming a late event is an error condition. p5 is a perfectly ordinary payment from a phone on a bad connection. Treating lateness as a fault means quarantining valid money.
Waiting for lag to become an outage before alerting on it. Nothing crashes while lag grows, so the first signal is a business user asking why the dashboard is an hour behind.
Better habit
Read a streaming metric by asking which of the two clocks it uses.
Before choosing a window type, say out loud whether one event should be able to count in two answers.
Watch lag as a trend, not as a threshold — a slow climb is the whole warning you get.
One event, three different answers
p5 never changes. It is the same payment, at the same moment, throughout. What changes is the window you read it through and how long you were prepared to wait for it — and those two choices alone decide whether your 12:00 total is 2 or 3.
Come back to this one
The next two topics name everything you just watched. If a definition there stops making sense, this timeline is where to return — the words are new, the behaviour is not.
Remember this
Six events, one of them late — and the count you report depends entirely on which window you read them through and how long you waited before answering.
Practice3 prompts
Find a combination where p5 corrects an already-published number, and say who would have seen the old one.
Explain, in your own words, why holding the clock back and keeping a window open are not the same dial.
In the overlapping view, pick any event that appears twice and explain why that is not double counting.
04 · Core technique
Windowing an Endless Stream
A window turns "the endless stream" into "this finite slice". Tumbling = fixed buckets, sliding = overlapping buckets, session = activity-bounded buckets.
⏱ 7 min · Topic 4 of 9
The three tables you just read have names. Fixed buckets are tumbling windows, overlapping ones are sliding windows, and the ones bounded by silence are session windows — that is the whole vocabulary, and you have already seen each behave.
Since you cannot wait for all the data, you aggregate over windows: finite slices of the stream. There are three classic types. Tumbling windows are fixed and non-overlapping ("events per minute"), so each event belongs to exactly one. Sliding windows are fixed-size but overlapping ("5-minute average updated every 30 seconds"), so an event can fall into several. Session windows are defined by gaps of inactivity, closing after a quiet period, perfect for grouping a user’s burst of activity.
Choosing the window type is choosing the question. "How many logins each hour?" is tumbling. "Rolling 5-minute error rate?" is sliding. "How long was each user’s browsing session?" is session. The example shows a tumbling per-minute count, the most common starting point.
Windows also bound state, which ties back to the previous topic: a tumbling 1-minute count only needs to remember the current minute, then it emits and forgets. This is how windowing keeps stateful aggregation from growing forever, the window is both the unit of aggregation and the lifetime of the state.
Core mental model
A window turns "the endless stream" into "this finite slice". Tumbling = fixed buckets, sliding = overlapping buckets, session = activity-bounded buckets.
Why it matters
Windowing is the defining technique of stream processing. The window type directly determines what your metric means, and a mismatch between the window and the question is a common, subtle source of wrong real-time numbers.
tumbling window
Fixed-size, non-overlapping time buckets; each event in exactly one.
sliding window
Fixed-size, overlapping buckets; an event can fall into several.
session window
A window bounded by a gap of inactivity, grouping bursts of activity.
window function
The aggregation applied per window (count, sum, average).
The same twelve events · three window shapesNot three ways of drawing the same answer. Three different questions.4 windows · 12 placements
Window
Events, in minutes past the hour
0.41.11.94.65.26.09.712.312.813.418.919.2
window
events
0–5 min
4
5–10 min
3
10–15 min
3
15–20 min
2
12 events, 12 placements — every event in exactly one windowTumbling windows partition time: no overlap, no gaps, and every event has exactly one home. This is the shape that maps onto a report — “revenue per five minutes” — and the one people mean when they say “windowed”.
Tumbling 1-minute count over a payment streamworked example
SQL
Input data
payments_stream (event_time)5 rows
payment_id
event_time
pa
12:00:10
pb
12:00:55
pc
12:01:05
pd
12:01:40
pe
12:01:59
-- Cut the endless stream into fixed 1-minute buckets and count.selectwindow_start,count(*)aspaymentsfrompayments_streamgroupbytumble(event_time,interval'1'minute);
payments per minute
window_start
payments
12:00
2
12:01
3
Two payments in the 12:00 window, three in 12:01. Each event lands in exactly one tumbling window, by its event time.
Tumbling windows partition time into fixed, non-overlapping buckets. Once the 12:00 window closes it emits its count and its state can be discarded.
Common mistake
Using a tumbling window when the question needs a rolling (sliding) one. You get disjoint per-bucket numbers instead of the smooth moving metric the question asked for.
Better habit
Match the window type to the question being asked.
Use windows to bound the lifetime of aggregation state.
State the window (type and size) when you define a streaming metric.
Production reality
A live operations dashboard often uses sliding windows (a smooth rolling rate), while billing uses tumbling windows (clean, non-overlapping periods). The same stream feeds both, with different windows.
Remember this
Windows slice an endless stream into finite buckets, tumbling, sliding, or session, and the window type defines what the metric means and bounds the state it needs.
Practice2 prompts
Pick a window type for: hourly logins, rolling 5-min error rate, user session length.
Explain how a tumbling window bounds aggregation state.
05 · The hard part
Event Time, Processing Time & Watermarks
Aggregate on event time (when it happened). A watermark estimates "we have seen everything up to T" so windows can close, trading a little latency for catching late data.
⏱ 6 min · Topic 5 of 9
That last column in the previous topic — newest event time seen, minus the wait — is called the watermark. You have already used it: it is the number that decided whether p5 counted.
Here is the deepest idea in stream processing. Every event has two timestamps: event time (when it actually happened, stamped on the device) and processing time (when your pipeline received it). They differ because of network delays, offline devices, and retries. Correct streaming aggregates on event time, otherwise a phone that was offline would have its midnight events counted at noon.
But event time creates a dilemma: if you aggregate "events in the 12:00 minute" by event time, when can you safely close that window? More 12:00 events might still be in flight. You cannot wait forever. The answer is the watermark: a moving estimate that says "I have probably now seen all events up to time T". When the watermark passes 12:01, you close the 12:00 window, accepting that a few stragglers may arrive after.
Watermarks make the trade-off explicit and tunable. A conservative watermark (wait longer) catches more late data but increases latency; an aggressive one is faster but drops more stragglers. Engines also offer allowed lateness, keeping a window open a bit longer to update it when late events trickle in. This is exactly the late-data problem from Chapter 12, now made precise for streams.
Core mental model
Aggregate on event time (when it happened). A watermark estimates "we have seen everything up to T" so windows can close, trading a little latency for catching late data.
Why it matters
Event time and watermarks are where streaming is most often gotten wrong, and where it most often silently miscounts. Understanding them is the difference between a real-time metric you can trust and one that is quietly off whenever the network hiccups.
processing time
When the pipeline received an event; easy but wrong for late data.
event time
When the event actually occurred; the correct basis for aggregation.
watermark
A moving estimate of how far event time has advanced, used to close windows.
allowed lateness
A grace period keeping a window open to update it with late-arriving events.
The 10:00–10:05 window · six events, arriving at their own paceA watermark is a claim that nothing older will arrive. Choose how long you are willing to wait to be wrong less often.closes 10 min past the hour
Allowed lateness
event time
arrived after
counted?
10:01
12 s
yes
10:02
24 s
yes
10:04
1.1 min
yes
10:04
7 min
dropped
10:03
42 min
dropped
10:01
25 h
dropped
3 of 6 counted · the answer is published 10 minutes past the hour3 events arrived after the watermark and were dropped. The watermark asserted that nothing older than 5 minutes would arrive; it was wrong, 3 times. That is not a bug in the watermark — being sometimes wrong is what a claim about the future is.
The answer to memoriseA watermark is not “where the stream has got to”. It is an assertion that no event older than this point will arrive, made by you, on a guess about the tail of the arrival distribution — and every window closes on that assertion. Interviewers ask this because most candidates describe what it does and cannot say what it claims.
Common mistake
Aggregating on processing time instead of event time. Late events are counted in the wrong window; offline-device data lands at the wrong time, skewing every period.
Setting watermarks too aggressively to reduce latency. Windows close before late-but-valid events arrive, silently dropping them from the counts.
Better habit
Aggregate on event time, with watermarks to close windows.
Tune the watermark and allowed-lateness to real observed delays.
Treat a stream metric’s lateness policy as part of its definition.
The processing-time trap
Using processing time is tempting because it never needs watermarks, but it makes every metric wrong whenever data is delayed. Production streaming aggregates on event time; the watermark is the price of correctness.
Interview note
Explaining event time vs processing time and how watermarks let windows close while tolerating lateness is one of the strongest things you can say in a streaming interview. It is the core of the discipline.
Remember this
Correct streaming aggregates on event time; a watermark estimates when all events up to a time have arrived so windows can close, trading latency against catching late data.
Practice2 prompts
Explain why an offline phone breaks processing-time aggregation.
Describe the latency-versus-completeness trade-off a watermark controls.
06 · In practice
Stream Joins, State & the Tools
Stream joins are windowed, stateful correlations. Flink for heavy stateful low-latency, Spark Structured Streaming for micro-batch in the Spark world, Kafka Streams for in-app Kafka-native processing.
⏱ 4 min · Topic 6 of 9
Real streaming jobs do more than count. Stream-to-stream joins correlate two streams within a time window (match each "click" to the "impression" that preceded it within 30 seconds). Stream-to-table joins enrich events with reference data (attach the user’s plan to each event). Both are stateful, the processor must hold recent events or the lookup table in managed, checkpointed state.
This is where the engines earn their keep. Apache Flink is the most powerful, with rich event-time semantics, large keyed state, and strong exactly-once via checkpointing, the choice for serious, stateful, low-latency streaming. Spark Structured Streaming uses a micro-batch model that reuses the Spark/SQL ecosystem, great when you already live in Spark and "seconds" is fresh enough. Kafka Streams is a lightweight library that runs inside your app, ideal for Kafka-native, per-service stream processing.
As everywhere in this course, learn the concepts, windows, event time, watermarks, keyed state, and the tools become choices rather than mysteries. An engineer who understands windowing and watermarks can read a Flink, Spark, or Kafka Streams job and follow exactly what it does.
Core mental model
Stream joins are windowed, stateful correlations. Flink for heavy stateful low-latency, Spark Structured Streaming for micro-batch in the Spark world, Kafka Streams for in-app Kafka-native processing.
Why it matters
Joins and enrichment are where streaming delivers real value (correlating events, adding context in real time), and they are deeply stateful. Knowing the tools’ trade-offs lets you pick the right engine instead of forcing a problem into a familiar one.
stream-to-stream join
Correlating two streams within a time window (clicks to impressions).
stream-to-table join
Enriching stream events with reference/lookup data.
Apache Flink
A powerful stream processor with rich event-time semantics and large managed state.
Kafka Streams
A lightweight stream-processing library that runs inside your own application.
Doing an unbounded stream-to-stream join with no time window. The join state grows forever waiting for matches; bound it with a time window.
Better habit
Bound stream joins with a time window to keep state finite.
Choose the engine by state needs and latency, not familiarity.
Learn windowing and watermarks once; apply them across all engines.
Production reality
Uber and Netflix run large Flink deployments precisely because their real-time use cases (pricing, quality of experience) need heavy, exactly-once stateful processing at low latency, the problem Flink is built for.
Remember this
Stream joins and enrichment are windowed, stateful operations; Flink, Spark Structured Streaming, and Kafka Streams implement the same concepts with different trade-offs in state, latency, and footprint.
Practice2 prompts
Explain why a stream-to-stream join must be windowed.
Pick an engine for a Kafka-native, per-service processor and justify it.
07 · Made real
The Same Streaming Job in Four Engines
Every streaming engine gives you the same four dials — when a window closes, how late is still counted, where state survives a restart, and how the result is written.
⏱ 8 min · Topic 7 of 9
One job: revenue per region, per five-minute window, tolerating two minutes of lateness, written where a dashboard can read it. That is the shape of most real streaming work.
Switch engine below. The job never changes — only what each one calls the window, the lateness bound, the state and the sink.
Core mental model
Every streaming engine gives you the same four dials — when a window closes, how late is still counted, where state survives a restart, and how the result is written.
Why it matters
Most streaming examples online stop at "read from Kafka and print". The parts that decide whether it works — the watermark, the state backend, the keyed sink — start where those examples end.
state backend
Where a job keeps per-key state between events. In memory it is fast and lost on restart; on RocksDB it survives and spills to disk.
savepoint
A deliberate, restorable snapshot of a job’s state, taken so you can deploy new code and carry the state across.
state TTL
An expiry on per-key state. Without a window or a TTL, a keyed stream accumulates keys for ever and the job eventually dies.
idle partition
A partition producing nothing. Its watermark stops advancing, which can hold back every window in the job until idleness is configured.
One job · revenue per region, per 5 minutes, 2 minutes of lateness allowedThe job never changes. Only the dialect does.micro-batch
ordersSOURCEOne event per order.→5-min windowPROCESSKeyed by region, event time.→live revenueSINKUpserted per window.
micro-batchRuns as a loop of tiny batches, so anything you know from batch Spark works. The trigger interval is the latency floor.
the idea
what this engine calls it
Lateness bound
withWatermark
Window
window(col, "5 minutes")
Fault-tolerant state
checkpointLocation
Custom sink write
foreachBatch
Spark Structured Streaming
from pyspark.sql.functions import col, from_json, window, sum as _sum
events = (spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", BROKERS)
.option("subscribe", "orders")
.option("startingOffsets", "latest")
.option("maxOffsetsPerTrigger", 500000) # bound each micro-batch
.load())
orders = (events
.select(from_json(col("value").cast("string"), ORDER_SCHEMA).alias("o"))
.select("o.*")
.withColumn("event_time", col("ordered_at").cast("timestamp")))
revenue = (orders
.withWatermark("event_time", "2 minutes") # how late is still counted
.groupBy(window(col("event_time"), "5 minutes"), col("region"))
.agg(_sum("total_gbp").alias("revenue_gbp")))
def publish(batch_df, batch_id):
# foreachBatch CAN run twice for the same batch_id — the write must be keyed
batch_df.selectExpr("window.start AS window_start", "region", "revenue_gbp") \
.write.format("org.opensearch.spark.sql") \
.option("opensearch.mapping.id", "window_start_region") \
.mode("append").save("live_revenue")
(revenue.writeStream
.outputMode("update") # emit windows as they change
.foreachBatch(publish)
.option("checkpointLocation", "s3://ckpt/live-revenue/")
.trigger(processingTime="30 seconds")
.start())
Four engines, four vocabularies, one job. When you meet a fifth, find its words for these four ideas and you can already read it.
Stream-to-stream join, and why it must be windowedworked example
SQL
-- Flink: match each payment to its order, if it arrives within an hourSELECTo.order_id,o.total_gbp,p.paid_atFROMordersoJOINpaymentspONo.order_id=p.order_idANDp.paid_atBETWEENo.ordered_atANDo.ordered_at+INTERVAL'1'HOUR;#Spark:thesameidea,andthewatermarksarewhatboundthebuffersorders_ws=orders.withWatermark("ordered_at","2minutes")payments_ws=payments.withWatermark("paid_at","2minutes")joined=orders_ws.join(payments_ws,expr("""order_id=payment_order_idANDpaid_at>=ordered_atANDpaid_at<=ordered_at+interval1hour"""),"leftOuter",#aleftOuteremitstheunmatchedrowonlyoncethewindow)#closes—soanorderwithnopaymentappearsanhourlate
Two live streams joined on a key. The interval is not a detail — it is what tells the engine when it may forget an unmatched order, and without it state grows until the job dies.
Enriching a stream from a slowly-changing tableworked example
SQL
-- Flink temporal join: use the FX rate that was current when the order happened,-- not the one that is current now. This is what "temporal" means.SELECTo.order_id,o.total*r.rateAStotal_gbpFROMordersoJOINfx_ratesFORSYSTEM_TIMEASOFo.ordered_atASrONo.currency=r.currency;#Spark:broadcastasmalldimensionandrefreshitperiodicallyregions=spark.read.parquet("s3://marlow-silver/regions/")#small,static-ishenriched=(orders_stream.join(broadcast(regions),on="region_id",how="left"))#noshuffle#ForadimensionthatCHANGES,astream-staticjoinre-readsthestaticside#eachmicro-batch—fineforSpark,andagenuinereasonpeoplepickit.
The most common real requirement, and the one that catches people: joining live events to reference data without re-reading the whole table per event, and getting the version that was true at event time.
Stateful processing with your own logicworked example
Windows cover most cases; sessionisation, alerting and state machines do not. Both engines let you keep arbitrary per-key state — and both make you say when to throw it away.
Running it: the operational settings nobody showsworked example
A streaming job is a service. These are the settings that decide whether it survives a bad day — and the checkpoint location is the one you must never casually delete.
Choosing an engine on what it actually costs you
Engine
Latency floor
Deploy as
Best when
The catch
Spark Structured Streaming
Trigger interval — seconds
A Spark job on your existing cluster
You already run Spark, and batch and streaming share logic
Micro-batch means sub-second is not on offer
Flink
Milliseconds
A long-lived cluster or app
Latency, complex state, or event-time correctness really matter
Its own runtime to learn and operate
Kafka Streams
Milliseconds
A library inside your service
Kafka in, Kafka out, owned by one service team
JVM only; scaling is scaling your service
Streaming SQL (Materialize, RisingWave)
Sub-second
Nothing — you write a view
The job is expressible as SQL over streams
Escaping SQL is hard when you need to
Cloud-managed (Dataflow, Stream Analytics)
Sub-second to seconds
A submitted job
You want no cluster and are on that cloud
Portability, and per-hour billing while idle
The four dials, in each engine’s words
Idea
Spark
Flink
Kafka Streams
How late is still counted
withWatermark
WATERMARK FOR … AS …
Grace period on the window
Window
window(col, "5 minutes")
TUMBLE / HOP / SESSION
TimeWindows.ofSizeAndGrace
State that survives restart
checkpointLocation
State backend + checkpoints
RocksDB + changelog topic
Writing somewhere custom
foreachBatch
A connector or SinkFunction
to(topic), then a connector
Upgrading without losing state
Compatible checkpoint
Savepoint + stable operator UIDs
Changelog replay
What actually breaks a streaming job in production
Symptom
Usual cause
What to change
Lag grows and never recovers
Throughput below the arrival rate
More parallelism, or bound the batch and fix the slow operator
Job dies with OOM after days
Unbounded state — no watermark or no TTL
Add a watermark, a window, or an explicit state TTL
Numbers change after a restart
Non-idempotent sink plus at-least-once replay
Key the sink write — Chapter 11
Restart replays everything
Checkpoint deleted or incompatible
Treat checkpoints as production state; use savepoints to upgrade
One partition lags, the rest are fine
Key skew — one region dominates
Change the key, or salt it and aggregate twice
Windows never emit
Watermark not advancing — an idle partition
Configure idleness so a quiet partition cannot hold the clock back
Common mistake
Deleting a checkpoint directory to "clear a stuck job". The job loses its offsets and its state, so it either replays the retention window or skips everything written while it was down.
Keying a stream by something unbounded, like session id, with no TTL. State grows for ever and the job dies days later, far from the change that caused it.
Assuming foreachBatch runs exactly once per batch. It can re-run the same batch id after a failure, so an unkeyed write inside it duplicates.
Changing a job’s operator graph without stable UIDs. The savepoint no longer maps to the new job, and the only way forward is to start from scratch.
Better habit
Put a watermark or a TTL on every piece of state, before it ships.
Treat the checkpoint as production data — back it up, never casually delete it.
Bound each batch or each fetch, so recovery after downtime cannot stampede.
Alert on consumer lag and on watermark age, not just on whether the job is running.
A streaming job is a service, not a script
It has uptime, a deploy story, state that must survive that deploy, and a pager. Teams that ship one as a script discover all four during the first upgrade.
The follow-up you will actually get
"Your streaming job has been running for a month and suddenly OOMs — why?" The expected answer is unbounded state: a keyed aggregation with no window, no watermark and no TTL, accumulating keys until the heap runs out.
Remember this
Every engine offers the same four dials — window, lateness, durable state, and how the result is written — so learn the dials once and the dialect becomes a lookup.
Practice2 prompts
Take a batch aggregation you own and write the streaming version, naming the window and the lateness you would allow.
For a streaming job you run, say where its state lives and what happens to that state during a deploy.
08 · Practice
Practice Lab
A watermark is a claim about completeness, not a measurement of time. You are asserting that nothing older than this will arrive, and you can be wrong.
⏱ 3 min · Topic 8 of 9
Five scenarios about the window and the watermark: when a stream is allowed to answer, and what you owe the records that arrive afterwards.
Build them on the pipeline canvas with the chapter closed. Each is graded against the scenario’s own requirement rules rather than against a model answer, so there is more than one design that passes — and a design that does not pass is told exactly which requirement it missed.
Core mental model
A watermark is a claim about completeness, not a measurement of time. You are asserting that nothing older than this will arrive, and you can be wrong.
Why it matters
Windowing and watermarks are the deepest material in the streaming half of this module and are asked by name wherever Flink runs. Each scenario below forces the completeness-versus-latency decision rather than letting you defer it.
Common mistake
Windowing on ingest time because it is convenient, so the buckets no longer mean what the report says they mean. The review will find it, but only after you have submitted a design you believed in — which is the point. That is the memory that survives interview pressure.
Revealing the reference design before your own review comes back. You see what correct looks like without finding out what your version got wrong, and your version is the one you will draw again under pressure.
Better habit
For every window, state its allowed lateness and what happens to a record that misses it. "It is dropped" is an acceptable answer only if you said it on purpose.
Run the review, fix what it finds, and run it again. The second score is the one that means something.
Narrate the finished design out loud in ninety seconds. If you cannot, the design has a hole you have not found yet.
In four of these the design changes depending on whether you publish on time with a hole or publish complete and late. Choose before you build.
These are the round, not a warm-up
Five graded design scenarios, each with staged requirement checks and interview probes of its own. Working them out loud, against a clock, is the closest rehearsal to the real thing this module offers.
Remember this
A chapter read is a chapter you can recognise; a scenario built and reviewed is one you can use. Close this and go build.
Practice2 prompts
Before opening any scenario, write down the freshness requirement and who consumes the output.
After each review, write one sentence naming the requirement you missed and why you missed it.
09 · Next Chapter
Next Chapter
You can now process endless streams with windows, event time, and state. But streams deliver events that can arrive out of order, more than once, or fail to process. The next chapter makes those guarantees precise.
⏱ 3 min · Topic 9 of 9
Next chapter
Delivery Guarantees & Ordering
You can now process endless streams with windows, event time, and state. But streams deliver events that can arrive out of order, more than once, or fail to process. The next chapter makes those guarantees precise.
Chapter 15 covers delivery guarantees and ordering: partition-level ordering, consumer groups, checkpointing, dead-letter queues, and achieving exactly-once inside a stream.
