Delivery Guarantees & Ordering

Crashes and retries happen in every stream. This chapter shows what each one does to your data, and how to keep the result right.

What you already know

• Kafka keeps events in a log. A topic is split into partitions, and each event has an offset.
• A committed offset is the offset of the next record to read. Lag is how far a reader is behind.
• Order exists only inside one partition. The same key goes to the same partition.
• The chapter Idempotency, Retries & Exactly-Once showed how to make a repeated write harmless.

What you will learn

1. At-most-once and at-least-once, as a crash on the shop stream.
2. What exactly-once really means.
3. How Kafka transactions work, and where they stop.
4. How to remove repeats with a time window.
5. What can break order inside a partition.
6. How to keep the newest version of a row when events arrive out of order.
7. What to do with a message that always fails.
8. Which guarantee to pick for which data.

We use the same shop and the same made-up day, 2026-01-22. This table shows where the orders from the last chapter are stored:

You met the three delivery promises in Idempotency, Retries & Exactly-Once. Now you see the first two on the shop stream, as a crash.

A consumer reads partition p0. For each order, it writes a row to the warehouse table orders_wh and commits the offset. The order of these two steps decides what a crash does.

This is partition p0. The correct total in orders_wh is 30.00 + 7.50 + 18.00 = 55.50:

# Loop 1: commit first, then write
for record in consumer:        # pseudocode: reads p0 in offset order
    consumer.commit(record)    # save "next offset to read"
    insert_into_orders_wh(record)

# Loop 2: write first, then commit
for record in consumer:
    insert_into_orders_wh(record)   # plain INSERT
    consumer.commit(record)

# The crash: the consumer stops while it handles order 114 (offset 1).

In loop 1, the commit for offset 1 saved "next is 2". The consumer restarted at order 150, so order 114 was never written. This is at-most-once: a message can be lost, but never repeated.

In loop 2, the last commit was for order 110, "next is 1". The consumer restarted at order 114 and wrote it again. This is at-least-once: a message is never lost, but it can be repeated.

Most pipelines choose at-least-once, because a lost order is worse than a repeated one. Then they make the repeat harmless. With a write by key, loop 2 ends with rows 110, 114 and 150, and a total of 55.50.

No setting can stop a crash from making the consumer read an order twice. So exactly-once cannot mean "each message is delivered once". It means the result is the same as if each message was handled once.

Engineers also call this effectively-once.

A system can promise this in two ways. Inside one system, such as Kafka to Kafka, the system saves the output and the offset together. End to end, from the topic to your warehouse, the sink must help too.

A sink is the place that receives the results. It helps in one of two ways.

• Idempotent: a repeated write changes nothing.
• Transactional: it saves the rows and the offset together, or not at all.

A sink that does neither will show repeats after a crash.

This table shows four setups for the topic orders. All of them read at-least-once:

Setup D is the loop 2 from the last section. Setups B and C work with any sink that can do a keyed write or a transaction. Setup A works only when the output is a Kafka topic.

Flink, another stream tool, has the same limit. Its checkpoints give exactly-once for its own state.

A sink outside Flink must join the checkpoint or be idempotent. The chapter Stream Processing explains checkpoints.

A Kafka transaction makes two steps one unit: write the results to another topic, and commit the input offsets. Either both happen, or neither does. The whole read, change and write loop is called consume-transform-produce.

Three settings make it work:

• The producer has a transactional.id, a name that stays the same when the app restarts.
• Readers of the output set isolation.level=read_committed.
• The input consumer turns auto-commit off.

This loop reads two orders from p1 and writes them to a new topic, orders_clean:

producer = Producer({"bootstrap.servers": BROKERS,
                     "transactional.id": "orders-clean-1"})
producer.init_transactions()

while True:
    records = consumer.consume(2, timeout=1.0)   # enable.auto.commit = false
    producer.begin_transaction()
    try:
        for r in records:
            producer.produce("orders_clean", key=r.key(), value=clean(r.value()))
        # The offsets are saved INSIDE the same transaction.
        producer.send_offsets_to_transaction(
            consumer.position(consumer.assignment()),
            consumer.consumer_group_metadata())
        producer.commit_transaction()
    except Exception:
        producer.abort_transaction()   # no output is shown, no offset moves
        raise

# A reader of orders_clean sets: isolation.level = read_committed

The offset moved only in step 4, together with the output. So the output holds each order once, even though the input was read twice.

In the Java client, the default is isolation.level=read_uncommitted. Such a reader would see 111 and 112 twice.

Sometimes the old app is not dead, only slow. When a new copy starts with the same transactional.id, Kafka blocks the old copy, a "zombie", from writing.

Transactions have a cost. A reader sees the output only after the commit, so results arrive a little later.

You can remove repeats in the stream itself. The job remembers each order_id it has seen. If an id comes again, the job drops it.

But a stream never ends, so the memory would grow for ever. So the job forgets an id after a time limit. This limit is the window.

Here the window is 5 minutes. An id is forgotten 5 minutes after it was first seen. The job counts orders:

seen = {}                          # order_id -> time first seen
count = 0
for event in stream:
    forget_ids_older_than_5_minutes(seen, now=event.arrives)
    if event.order_id in seen:
        continue                   # a repeat: drop it
    seen[event.order_id] = event.arrives
    count += 1

At 10:15 the job had forgotten everything: 113 at 10:12, 114 at 10:13 and 150 at 10:14. So the late repeat looked new.

The window is a trade. A longer window catches later repeats, but the job needs more memory. Make the window longer than the longest gap you expect between a message and its repeat.

Order exists only inside one partition. A reader of p0 sees orders 110, 114 and 150 in that order, every time. Three things can break even that.

Break 1: a retry passes a newer batch

A producer can send a second batch before the first one is confirmed. Kafka calls this setting max.in.flight.requests.per.connection. Say it is 2, and the producer has no idempotence.

Say customer 2 sends two orders to an empty partition. The producer sends batch A with order 110 and then batch B with order 150:

Now a reader sees order 150 before order 110, but customer 2 sent 110 first. With enable.idempotence=true, the broker uses sequence numbers to keep the order, with up to 5 batches in flight. Or set max.in.flight.requests.per.connection to 1, which is slower.

Break 2: you add partitions

The key picks the partition with a rule that uses the number of partitions. If that number changes, the same key can go to a different partition. Old events stay where they are.

Our made-up rule is customer_id mod 2. With 3 partitions it becomes mod 3. This table shows where each customer goes:

Look at customer 2. Orders 110 and 150 are in p0. A new order 170 for customer 2 goes to p2.

A reader of p0 and a reader of p2 work at their own speed. Order 170 can be handled before order 150.

Kafka cannot move old events. So plan the number of partitions before you start, and add partitions only when you accept this.

Break 3: two producers

The log keeps the order in which the broker received the events. Two producers can write the same key, and the log order can differ from the real order.

Say server X sends "order 172 placed" at 10:12. Server Y sends "order 172 paid" at 10:13, but its message arrives first. The log reads paid, then placed.

The next section shows how to cope.

Sometimes you cannot stop events from arriving out of order. Then you must make the result right anyway. The idea is simple: give every update a version, and keep only the highest one.

Here two servers send status updates for orders 172 and 173. Each update has a version number.

The source database raises the version by one at each change. The updates arrive in this order:

Update 3 is old. Order 172 was placed at 10:12, before it was paid at 10:13. But it arrived after the paid update.

The simple rule is "the last write wins": the last update to arrive replaces the row. The better rule is to write only when the new version is higher:

INSERT INTO order_status (order_id, status, version)
VALUES (172, 'placed', 1)
ON CONFLICT (order_id) DO UPDATE
   SET status = EXCLUDED.status, version = EXCLUDED.version
 WHERE EXCLUDED.version > order_status.version;

With the plain "last write wins" rule, order 172 would end as placed. That is wrong, and nothing warns you.

A timestamp can also be the version, but be careful. Two updates can have the same updated_at, so the rule cannot pick one. The clocks of two servers can also differ.

A version number that one place raises for each change never ties. Another idea is a sequence number per key. A gap in the numbers, such as 1 then 3, tells you that update 2 is late or missing.

For counts by time, you do not need versions. Use event time with windows and watermarks, as in the chapter Stream Processing.

A poison message is a message that fails every time you process it. Say order 175 arrives with the amount "N/A". The code cannot read a number, so it throws an error.

At-least-once says: do not skip data. So the consumer retries and fails, again and again. It never commits past the bad record, so everything behind it waits.

Order 175 and a good order 176 arrive in partition p1, after order 151. The log of p1 now ends at offset 6:

The committed offset stays at 4. Lag is the log end offset minus the committed offset: 6 - 4 = 2. It grows with every new order, and the stream goes stale.

The fix is a dead-letter queue, or DLQ: a separate topic for messages that failed. The consumer tries a fixed number of times, for example 3. Then it saves the message in the DLQ and moves on.

def handle(record):
    for attempt in range(1, 4):
        try:
            process(record)
            return
        except ValueError as e:
            last_error = e
    # all 3 tries failed: save the ORIGINAL message and why it failed
    dlq.produce("orders_dlq", key=record.key(), value=record.value(),
                headers={"error": str(last_error)[:200],
                         "source_partition": str(record.partition()),
                         "source_offset": str(record.offset())})
    dlq.flush()           # make sure the DLQ copy is saved first

for record in consumer:
    handle(record)
    consumer.commit(record)   # only now move past order 175

A good DLQ record keeps the original message and the reason. With the source partition and offset, you can find the record again. After you fix the code, you read the DLQ and send the messages back.

Watch the number of DLQ messages per hour. A few are normal. A sudden jump means something broke.

A stronger promise needs more work. Pick the weakest promise that is still safe for the data. This list shows what each choice costs:

• At-most-once: the cheapest. You commit first and accept gaps.
• At-least-once with a keyed write: every message needs a key, and the sink must support a keyed write.
• Kafka transaction: results arrive a little later, and it works only from Kafka to Kafka.
• Transactional sink: you must save the offset in the same database as the rows.

This table picks a guarantee for four kinds of data from the shop:

You rarely need more than at-least-once and a keyed write. Use a Kafka transaction when the output is another topic.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.