Kafka in Production

Kafka is the most common log for streaming data. Earlier chapters showed how it works on one good day. This chapter shows how teams run it for years, when brokers fail and code changes.

Interviews for senior roles ask about this part. They want to hear what you set, what it protects, and what it still cannot protect.

What you already know

• A topic is a log. A partition is one ordered part of it. Each record has an offset.
• The key picks the partition. Order exists only inside one partition.
• A consumer group shares the partitions. Each group keeps its own committed offsets.
• Lag is the log end offset minus the committed offset.
• acks=all and an idempotent producer make writes safer. A Kafka transaction gives exactly-once from Kafka to Kafka.

If a line in this list is new to you, read "Streaming Architecture Fundamentals" and "Delivery Guarantees & Ordering" first.

What you will learn

1. How copies of a partition keep data safe, and when they do not.
2. How to choose the key and the number of partitions.
3. Why a deploy can stop a whole consumer group, and how to stop that.
4. Compaction: keeping the latest value for each key.
5. Kafka Connect: moving data in and out without writing a consumer.
6. How to roll out a schema change to many consumers.
7. How to replay a few hours for one group only.
8. How Kafka compares with Kinesis, Pub/Sub and Event Hubs.

We use the same shop and the topic orders. All numbers are made up and small, so you can check every row.

A broker can die. So Kafka keeps copies of each partition on different brokers. The number of copies is the replication factor.

One copy is the leader. Producers write to it and consumers read from it. The other copies are followers, and they copy the leader all the time.

Here the topic orders has replication factor 3, on brokers b1, b2 and b3. For partition p0, b1 is the leader.

The ISR: the copies the leader trusts

A follower that keeps up with the leader is in sync. The list of in-sync copies, with the leader, is the ISR.

A follower that falls behind for longer than replica.lag.time.max.ms leaves the ISR. The default is 30 seconds. It comes back when it catches up.

A consumer can read a record only when every copy in the ISR has it. The first offset that not all of them have is the high watermark.

This table shows p0 at one moment. All three copies are in the ISR:

The smallest log end offset in the ISR is 4, so the high watermark is 4. Consumers can read offsets 0 to 3.

Offset 4 exists on b1 and b2, but consumers wait until b3 has it too. So a consumer never reads a record that a leader change could lose.

What acks=all really waits for

With acks=all, the leader answers "done" when every copy in the ISR has the record. It does not wait for all three copies, only for the ISR.

If the ISR has shrunk to the leader alone, "all" means one copy. The setting min.insync.replicas stops that.

It is the smallest ISR that may accept an acks=all write. Its default is 1. This table shows a topic with min.insync.replicas=2:

In the last row Kafka refuses the write on purpose. One disk is not safe enough, so the producer retries and then reports an error.

With the default of 1, the same write would be accepted on b1 alone. If b1 then failed, the order would be lost.

When every ISR copy is gone

Sometimes every copy in the ISR fails, and only an old follower is left. That follower is missing recent records.

By default Kafka waits, and the partition is offline until an ISR copy comes back. The setting is unclean.leader.election.enable=false.

If you set it to true, the old follower becomes leader. The partition comes back sooner, but the records it never got are gone for good.

Two choices are hard to undo: the key and the number of partitions. Make them on purpose, before the topic is busy.

The key decides the order

Use the key that must stay in order. If each customer's orders must stay in order, the key is customer_id.

A key gives order for that key and nothing more. Two customers on different partitions have no order between them.

With no key, records are spread over the partitions. That is fine for clicks you only count, but not for changes to one order.

The count decides the speed

Inside one group, one partition is read by one consumer. So the partition count is the most consumers that can work at once.

Size it for the peak you expect later, not for today. This is the arithmetic for the topic orders:

Measure "what one loader can do" with a test. The slow part is usually the sink, not Kafka.

Do not choose 1,000 "to be safe". Every partition adds files, copies to send and work when a broker fails.

Adding partitions later moves keys

You can add partitions, but you can never remove them. Adding them changes where most keys go.

Kafka picks the partition as hash(key) mod the number of partitions. To keep it easy, we use customer_id mod the count:

-- Made-up rule, so you can check each row:
--   before: partition = customer_id mod 2
--   after:  partition = customer_id mod 3
-- Real Kafka uses hash(key) mod count. Most keys move in the same way.

Say customer 2 has 1,000 unread orders in p0. Its next order goes to the empty p2 and is read first. For a job that tracks order status, that is wrong data, not just a delay.

If you must grow a keyed topic, pause the producers, let the consumers finish, and then add partitions. Or create a new topic and move to it at a known point.

One key is too hot

A hot key is one key with far more records than the rest. Say customer 9 is a big seller with 30 percent of all orders.

Its partition gets 30 percent of the load, and one consumer cannot keep up. The fix is to split only that key:

def key_for(order):
    if order.customer_id in HOT_CUSTOMERS:          # for example {9}
        bucket = order.order_id % 4
        return f"{order.customer_id}-{bucket}"      # "9-0" to "9-3"
    return str(order.customer_id)                    # every other customer

You give up one thing: the orders of customer 9 are no longer in one line. A job that needs a total for customer 9 must add the four buckets.

A rebalance is when a consumer group shares out its partitions again. It happens when a consumer joins, leaves, or is thought to be dead.

A deploy restarts every consumer, one after another. So a deploy can cause many rebalances.

Here the group loader has 4 pods and 8 partitions, so each pod reads 2 partitions.

Three ways a restart can go

The older way is eager rebalancing. Every pod gives back all its partitions, and the whole group waits for the new plan.

Cooperative rebalancing, since Kafka 2.4, moves only the partitions that must move. The other pods keep reading.

Static membership gives each pod a fixed name, set with group.instance.id. If the pod comes back within session.timeout.ms, it gets its partitions back with no rebalance.

This table compares the three during a deploy of all 4 pods:

The default session.timeout.ms is 45 seconds. A pod that takes 2 minutes to start is treated as gone, even with a fixed name.

In Kafka 4.x the default assignor list still uses the eager range assignor. Moving a running group to cooperative takes two rolling deploys.

Kafka 4.0 also added a new group protocol, where the broker plans the assignment. Clients turn it on with group.protocol=consumer.

A slow batch also causes a rebalance

A consumer must call poll() again within max.poll.interval.ms. The default is 5 minutes. If a batch takes longer, Kafka removes the consumer from the group.

Its partitions go to another consumer. That consumer starts from the last committed offset, so the slow batch is read again.

So a group with slow batches keeps rebalancing and writes duplicates. Fix the batch: take fewer records per poll with max.poll.records, or make the sink faster.

Some topics hold the current state of things, such as each customer's address. Old addresses do not matter, but the latest one must stay for ever.

Time-based retention would delete the address of a customer who never moves. Compaction solves this. It keeps at least the latest record for each key and removes older ones.

You turn it on per topic with cleanup.policy=compact. A record with a key and no value is a tombstone. It means "delete this key".

Here is the topic customer_address. Offsets 0 to 4 are in closed segments, the old files of the log. Offset 5 is in the active segment, the file Kafka is writing now:

# Topic setting
cleanup.policy=compact

# Compaction cleans closed segments only.
# The active segment (here, offset 5) is never compacted.

Look at c1. It still has two values, Munich and Hamburg, because offset 5 is in the active segment. Compaction happens later, so a reader can still see old values.

The tombstone at offset 4 stays for delete.retention.ms, one day by default. Then it is removed too.

A consumer that is offline longer than that can miss the delete. It would keep customer c2 for ever.

Compacted topics are used for database changes from CDC, for state saved by stream jobs, and by Kafka itself. Committed offsets live in the compacted topic __consumer_offsets.

Most data in Kafka comes from a database or goes to storage. Kafka Connect does this without a consumer you write yourself.

You send it a configuration, and it runs the job. A source connector reads a system into Kafka. A sink connector writes Kafka into a system.

Connect has three parts:

The cluster saves its configurations, offsets and status in three Kafka topics. So if a worker dies, another worker carries on from the saved offset.

A source: Postgres into Kafka with Debezium

Debezium is a set of source connectors for CDC, change data capture. This one reads every change to two Postgres tables:

{
  "name": "shop-db",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "shop-db.internal",
    "database.dbname": "shop",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_shop",
    "topic.prefix": "shop",
    "table.include.list": "public.orders,public.customers"
  }
}

This connector reads one replication slot, which is one ordered stream. So it always runs one task, whatever tasks.max says.

The slot is the real risk. While the connector is stopped, Postgres keeps the log it has not read, and the database disk fills up. Alert on slot lag.

A sink, and a place for bad records

A sink can meet a record it cannot convert. By default the task stops, and nothing more is written.

For a sink you can send bad records to a dead-letter topic instead. This S3 sink writes Parquet files and sets one:

{
  "name": "orders-to-s3",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "shop.public.orders",
    "s3.bucket.name": "shop-lake-bronze",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "flush.size": "10000",
    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq.orders-to-s3",
    "errors.deadletterqueue.context.headers.enable": "true"
  }
}

This dead-letter topic works only for sink connectors. A source connector fails before a Kafka record exists, so there is nothing to send.

Connect or your own consumer?

A schema is the shape of a record: its fields and their types. The chapter "Schema Evolution & Data Contracts" teaches which changes are safe. Here you see how Kafka teams roll one out.

Most teams keep schemas in a Schema Registry. It is a separate service. The Kafka broker never checks schemas, it only stores bytes.

The producer's serializer puts a small schema ID in front of each record. The consumer uses the ID to fetch the schema that wrote the record.

The registry checks a new schema when it is registered. The rule is set per subject, usually one per topic. The rule decides who must upgrade first:

A rule with _TRANSITIVE at the end checks every earlier version, not only the last one.

Here is a real kind of break. The topic orders has 12 consumer teams and the default BACKWARD rule.

The producer team renames the field amount to total, and gives total a default. A rename is a delete plus an add.

The registry accepts it, because a new reader can read old records: it fills in the default. So the producer deploys first and stops writing amount.

Old consumers still expect amount, and their schema has no default for it. They cannot read the new records, and they fail.

BACKWARD only promised that new readers can read old records. It said nothing about old readers and new records, which is what broke.

The safe way to rename amount to total has four steps:

1. Add total with a default. Keep amount. The producer writes both.
2. Move each of the 12 consumers to total, one team at a time.
3. Check that no consumer reads amount any more.
4. Remove amount in a new version.

A bug in the warehouse loader wrote wrong amounts from 03:00 to 09:00. The fix is deployed. Now the loader must read those six hours again.

Other groups, such as the fraud job, read the same topic. They must not be touched. Offsets belong to each group, so you can move one group alone.

First stop every consumer in the group. Kafka does not reset the offsets of a group with running members. Then preview the reset with --dry-run:

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group loader --topic orders \
  --reset-offsets --to-datetime 2026-01-22T03:00:00.000 --dry-run

# The output looks right? Run the same command with --execute.

Two things must be true first:

• The six hours are still inside retention. With 7 days of retention, they are.
• The timestamps mean what you think. By default a record carries the producer's clock, and a wrong clock moves the reset point.

The sink decides if the replay fixes the data or doubles it. This table shows the same replay with two loaders:

If the loader cannot stop, run a second group with its own group.id. It reads from 03:00 to 09:00 and then stops, while the live group carries on.

Each cloud has its own streaming service. The ideas are the same: a log, an order key, and a time limit. The limits and the work differ.

This table compares them. The numbers come from each vendor's documentation in October 2026, and they change, so check them again:

Choose Kafka when you need Kafka Connect, Kafka transactions, compacted topics, or long history that many groups replay.

Choose the cloud service when you want no brokers to run, and its limits fit. Then say which limit you checked.

Two more facts help in this talk. Kafka 4.0 runs without ZooKeeper, in KRaft mode. Since Kafka 3.9, tiered storage can move old segments to object storage, so long retention costs less disk.

Senior Kafka questions are rarely "what is a partition". They give you a problem and ask what you would set. This table lists common ones, the short answer, and the card in the question bank:

In every answer, say the setting, what it protects, and what it still cannot protect. That last part is what makes the answer senior.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.