What You'll Master Here
You cannot prevent retries, so make them harmless. The goal is not "never process twice"; it is "processing twice has the same effect as processing once".
A customer pays £120. Your consumer writes it to the warehouse and then dies, a second before it can record that it did. The queue, having heard no acknowledgement, delivers the payment again. Tomorrow finance sees £240 and nothing anywhere reported an error.
That gap — between the data being safe and the queue knowing it is safe — cannot be closed. This chapter is about making it harmless, which is the correctness problem underneath every reliable pipeline you have studied so far.
We build it from the ground up. First, delivery semantics, the three honest options for what happens to a message when things go wrong: at-most-once, at-least-once, and exactly-once. Then idempotency, the property that makes re-running safe. Then the deduplication keys that implement it, and finally the honest truth about "exactly-once".
By the end you will understand why the pragmatic industry recipe is "at-least-once delivery plus idempotent processing", be able to make any write safe to retry, and be able to give the mature, slightly contrarian answer to "does your pipeline guarantee exactly-once?", which is what senior engineers actually say.
You cannot prevent retries, so make them harmless. The goal is not "never process twice"; it is "processing twice has the same effect as processing once".
Retries are not an edge case; they are constant. Schedulers retry, networks retry, consumers retry. A pipeline that is not safe under retries will eventually double-count revenue or lose an event. This chapter is the difference between a pipeline that is correct only on a good day and one that is correct always.
- delivery semantics
- The guarantee about how many times a message is processed: at-most, at-least, or exactly once.
- idempotency
- An operation that, applied repeatedly, has the same effect as applying it once.
- retry
- Re-attempting a failed operation; inevitable, so it must be safe.
- deduplication
- Removing repeated effects of the same logical event, usually via a unique key.
Designing as if each event is processed exactly once by default. The first crash-and-retry double-applies an event, and the bug hides until the numbers drift.
Assume every operation may run more than once.
Make writes idempotent so retries are safe.
Pair at-least-once delivery with deduplication on a key.
Stop trying to guarantee "process once". Instead make "process again" harmless. Idempotency turns the impossible (perfect once-only delivery) into the achievable (safe repetition).
Read the semantics topic first to get the vocabulary, then idempotency, then the honest-truth topic. The recipe "at-least-once + idempotent" is the thread through all of it.
Retries are constant, so correctness comes from making repetition harmless: at-least-once delivery plus idempotent processing is the practical recipe for exactly-once results.
- Define idempotency in one sentence.
- Explain why "never process twice" is the wrong goal.