What You'll Master Here
A stream guarantees order only within a partition and, by default, at-least-once delivery. Everything stronger (global order, exactly-once) is engineered on top, with real trade-offs.
A stream promises to move every event from producer to consumer, but the real world, crashes, network failures, retries, makes that promise surprisingly hard to keep precisely. This chapter is about the guarantees a streaming system actually offers around two questions: in what order will events arrive, and how many times will each be processed?
You already met the delivery-semantics vocabulary in Chapter 11; here you will see how it plays out inside a real streaming platform, through partition ordering, offset commits, checkpoints, and dead-letter queues. These are the concrete mechanisms behind "exactly-once" claims in Kafka and Flink.
By the end you will be able to reason precisely about ordering (it is per-partition, never global), explain how streaming systems approach exactly-once with checkpoints and transactions, and design for the poison message that fails forever, the one that takes down naive consumers.
A stream guarantees order only within a partition and, by default, at-least-once delivery. Everything stronger (global order, exactly-once) is engineered on top, with real trade-offs.
Ordering and delivery guarantees are where streaming correctness lives. Misunderstand them and you build consumers that miscount on reorders, double-process on retries, or stall forever on a single bad message. These are the failures that page on-call engineers at 3 a.m.
- ordering guarantee
- The promise about the sequence events are delivered in; in streaming, per-partition only.
- delivery guarantee
- How many times an event is processed: at-most, at-least, or exactly once.
- checkpoint
- A saved snapshot of processing state and positions, used for exactly-once recovery.
- dead-letter queue
- A side channel for messages that fail processing repeatedly, so they do not block the stream.
Assuming a streaming platform gives global ordering and exactly-once for free. You design logic on guarantees that do not hold, and it silently miscounts or duplicates under failure.
Reason about ordering per partition, never globally.
Assume at-least-once and make consumers idempotent.
Plan for the message that fails forever.
Order is per-partition; delivery is at-least-once by default. Strong guarantees (global order, exactly-once) are built deliberately, at a cost. Know what you actually have before you rely on it.
Read ordering first, then delivery semantics in the streaming context, then exactly-once mechanics, then dead-letter queues. Each addresses one way streams break.
Streaming guarantees per-partition ordering and at-least-once delivery by default; this chapter is how stronger guarantees are engineered and how to handle messages that never succeed.
- State the default ordering and delivery guarantees of a partitioned stream.
- Define a dead-letter queue and the problem it solves.