Stream Processing

In "Streaming Architecture Fundamentals", producers wrote events to a log and consumers read them. This chapter is about what a consumer computes from those events.

A stream never ends. So you cannot say "count all the orders". You need windows, and you need a way to remember things between events.

What you already know

• Event time is when something happened. Processing time is when your job sees it.
• A late event arrives after events that happened later than it.
• A tumbling window is a fixed block of time, such as five minutes, with no overlap.
• A watermark is the newest event time seen, minus a wait. A window closes when the watermark reaches its end.
• A consumer reads events from a log, and its offset marks its place.

What you will learn

1. Steps that remember (stateful) and steps that do not (stateless).
2. Three kinds of window: tumbling, sliding and session.
3. How the remembered data, called state, grows and shrinks.
4. What a job can do with a late event.
5. How to join a stream to a table, and a stream to a stream.
6. How a checkpoint saves state, and how three engines name these ideas.

We keep the five shop orders from "Batch vs Streaming vs Micro-Batch", on the same made-up day, 2026-01-22. Look at both times for each order:

Order 113 is the late one. It was placed at 10:04, but it arrives at 10:07.

A stateless step needs only the event in front of it. A stateful step must remember earlier events.

A filter is stateless. This query keeps only the orders of 10.00 or more:

SELECT order_id, amount
FROM   order_events
WHERE  amount >= 10.00
ORDER  BY order_id;

A running total is stateful. The job must remember the total so far. That memory is called state.

This query adds up the revenue in the order the orders arrive:

SELECT order_id, amount,
       SUM(amount) OVER (ORDER BY arrives) AS running_total
FROM   order_events
ORDER  BY arrives;

Most state is kept per key. A key is the field you group by, such as customer_id. Each key has its own small piece of state.

Events with the same key go to the same worker. So one worker never needs the state of another key.

• Stateless steps are easy to scale and restart. Any worker can handle any event.
• Stateful steps need storage. If a worker dies, its state must come back.

A window cuts the stream into finite pieces by time. You count inside each piece. Here are three common kinds, all by event time.

Tumbling: fixed, no overlap

You saw this kind already. With five-minute windows, each order is in exactly one window:

Sliding: fixed, overlapping

A sliding window has a size and a slide. Size 10 minutes and slide 5 minutes give a new window every five minutes. Each order falls in two windows:

Order 110 is in the first two windows. Order 114 is in the last two. Use this kind for a moving view, such as "orders in the last 10 minutes, updated every 5".

Session: ends after a quiet gap

A session window groups events that are close together. It ends when no event comes for a set gap. So its length is not fixed.

Say the gap is 3 minutes, with one session for the whole shop and no key. Order 161 is new: customer 3, 25.00, placed at 10:20.

In session 1, every gap between events is 2 minutes or less, so the orders stay together. From 10:08 to 10:20 is 12 minutes, so order 161 starts a new session.

Window counts are state too. The job keeps one counter for each open window. Watch the counters as the five orders arrive, with a watermark wait of 3 minutes:

At 10:07, the late order 113 raises the 10:00 counter from 2 to 3. At 10:08, the watermark reaches 10:05. The job sends the final count and deletes the 10:00 counter.

Deleting is the point. Without a watermark, the job cannot know that a window is done. It would keep every counter for ever, until memory or disk runs out.

There are two ways to limit state:

• A watermark closes windows, and the job deletes their counters.
• A time to live (TTL) deletes the state of a key that has had no events for a set time.

A watermark is a claim: no event older than this will come. The claim can be wrong. An event that arrives after its window closed is too late.

Order 160 is new: customer 6, 15.00, placed at 10:03. It arrives at 10:09. At 10:09 the watermark is 10:05, so the 10:00 window is already closed.

The job has three choices for order 160. This table shows what each one gives for the 10:00 window:

The revenue of 51.00 is orders 110, 111 and 113 (30.00 + 9.00 + 12.00). With order 160 it is 66.00. Dropping makes the report wrong by one order.

Update gives the right number. But the job keeps the counter longer, and anyone who uses the result must accept that it can change. A MERGE on the window key can apply it.

A longer wait catches more late events. But the final answer comes later, and state lives longer.

In Flink, keeping a window open longer for late events is called allowed lateness.

A stream and a table

A stream-table join adds facts from a small table to each event. The job keeps a copy of the table, or looks it up. This query adds a country to each order:

SELECT o.order_id, o.amount, c.country
FROM   order_events o
JOIN   customers c ON c.customer_id = o.customer_id
ORDER  BY o.order_id;

In a stream job, each order is usually joined when it arrives, with the table as it is at that moment. The table can change later. An old result is not corrected.

Two streams

A stream-stream join matches events from two streams, such as orders and payments. Either event can arrive first. So the job keeps each event in state and waits for its partner.

Without a limit, the job would keep every event in state for ever. So you add a time bound: a payment counts only if it comes within 5 minutes after the order was placed. This query uses that bound:

SELECT o.order_id, o.amount, p.paid_at
FROM   order_events o
JOIN   payments p
  ON   p.order_id = o.order_id
 AND   p.paid_at BETWEEN o.event_time
                     AND o.event_time + INTERVAL '5 minutes'
ORDER  BY o.order_id;

Order 114 has no payment yet, so it is not in the result. A bound that is too tight loses real matches. A bound that is too loose keeps more state.

After the bound has passed, the watermark lets the job delete the order from state. To list unpaid orders, you need a left join. The engine can say "no payment" only after the watermark passes the bound, so those rows come out late.

State lives in a worker. If the worker dies, its state dies with it. A checkpoint is a saved copy of the state, together with the job's place in the input.

Here the job keeps the running total. It saves a checkpoint after order 111, then crashes:

The state and the position are saved together. So order 112 is not counted twice, and it is not lost. In step 6 the job reads order 112 again, but the total went back to before it.

• A bigger state makes each checkpoint slower. Keep state small.
• More checkpoints mean less to read again after a crash. But they cost more work.

Results already sent to other systems between step 2 and step 4 can be sent twice. That is the topic of the next chapter, "Delivery Guarantees & Ordering".

The ideas are the same in every engine. Only the names change. This table shows how three engines handle them:

Choose by where your data and your team already are. A Kafka-only Java service can use Kafka Streams. A team that already runs Spark can use Structured Streaming.

For Spark code, read "Structured Streaming Fundamentals" and then "Event-Time, Windows, Watermarks & Stateful Streaming" in the Apache Spark section.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.