Apache Kafka Theoretical

The reasoning

A topic is a name. The data lives in its partitions, each a sequence of records appended in order and never modified. Kafka's broker config documents `num.partitions` at 1 for a new topic, which is why production topics are created explicitly with more.

Within one partition, records are strictly ordered, and the offset is that order: record 0, record 1, and so on. Two partitions of the same topic have their own independent offset sequences, so offset 500 in partition 0 and offset 500 in partition 1 are unrelated records.

A consumer tracks one offset per partition it reads, the position of the next record to fetch. Committing an offset stores that number so a restart can resume. Reading does not delete anything; the offset just moves forward.

Because order is per partition, any ordering guarantee Kafka gives is per partition. Parallelism is bounded by the partition count, because one consumer in a group reads a partition at a time.

The reasoning

Each partition is copied to `replication.factor` brokers. With 3, a partition survives two broker failures. With the documented default of 1, there is one copy and a broker loss loses the partition until the broker returns.

Producers write to the leader and consumers read from it by default. Followers replicate by fetching from the leader like a consumer would. A follower that keeps up within `replica.lag.time.max.ms`, documented at 30 seconds, is in the ISR, the in-sync replica set.

When a leader fails, the controller elects a new leader from the ISR. If no in-sync replica exists, the partition goes offline unless `unclean.leader.election.enable`, documented as false, is turned on, which trades data loss for availability.

Replication is also what `acks=all` waits for. The leader acknowledges a write only when every ISR member has it. `min.insync.replicas` sets the floor; it is documented at 1 and set to 2 in production.

The reasoning

Kafka is a log, not a queue. A consumed record stays until retention removes it, so a second consumer group can read the same data and a failed consumer can replay. Retention is the only thing that frees disk.

A partition on disk is a directory of segment files. The active segment receives appends; when it reaches `log.segment.bytes` or `log.segment.ms`, it is closed and a new one opens. Each segment has index files for offset and timestamp lookup.

Retention checks closed segments. A segment is deleted when its newest record is older than the time limit or when the partition exceeds the size limit. The active segment is never deleted, so a quiet partition can hold records past the retention time.

`log.cleanup.policy`, documented as `delete`, chooses time or size deletion. Setting it to `compact` keeps the latest record per key instead, which is log compaction, the policy for changelog topics.

The reasoning

One request per record would be slow. The producer buffers records in memory, grouped by partition, and sends a batch per partition in one request. Bigger batches mean fewer requests, better compression, and higher throughput.

`batch.size` caps each batch in bytes. A record larger than it is sent on its own. Raising it to 64 KB or 256 KB lets hundreds of 1 KB records travel together. The memory for all batches comes from `buffer.memory`, documented at 32 MiB.

`linger.ms` is a deliberate delay. With 0, the producer sends as soon as a sender thread is free, so batches at low rates hold one record. With 5 to 20 ms, batches fill before they leave. The producer docs say the default is now 5 ms; it was 0 in older releases.

The two interact: a batch leaves when either limit is hit. At high rates `batch.size` governs; at low rates `linger.ms` does. Compression, `compression.type` documented as `none`, applies per batch, so bigger batches compress better.

The reasoning

Compression works on the batch, not the record. That is why it pairs with `linger.ms` and `batch.size`: a batch of 500 similar JSON events compresses several times better than one event.

The broker does not decompress by default. It validates the batch and appends it as is, so compressed data costs less disk and less replication bandwidth too. The broker-side `compression.type` can override this to recompress, which doubles CPU and is rarely wanted.

Consumers receive the compressed batch and decompress in the client. So the CPU cost sits on producer and consumer, and the network and disk savings sit everywhere in between.

Choosing a codec: snappy and lz4 for latency-sensitive paths, zstd for the best ratio at acceptable CPU, gzip when a consumer cannot use anything newer. Changing the codec needs no broker change, because each batch records its own.

The reasoning

Keyed records are about ordering. All events for `order_42` hash to the same partition, so they are consumed in the order they were produced. The cost is that the key's traffic cannot spread: one very busy key is one hot partition.

Keyless records only need throughput. The old behaviour was round-robin per record, which spread one producer's records across every partition and produced tiny batches everywhere. The sticky partitioner, default since 2.4, keeps sending to one partition until `batch.size` bytes are produced to it, then rotates, so batches fill and requests shrink.

The producer docs add that `partitioner.adaptive.partitioning.enable`, documented as true, lets the producer prefer partitions on faster brokers, so a slow broker does not hold back the whole producer.

Changing the partition count changes the modulo, so a key that hashed to partition 3 of 12 may hash elsewhere with 48 partitions. Old records stay where they are, which breaks per-key ordering across the change.

The reasoning

A record larger than `max.request.size` is rejected by the producer before it leaves. One that passes but exceeds the broker's `message.max.bytes`, or the topic-level override, is rejected by the broker. A consumer whose `max.partition.fetch.bytes` is smaller than a record cannot make progress past it.

Kafka is tuned for many small records. Raising all three limits to 10 MB works but costs memory on every client and broker and makes batches lumpy. Compression counts after the fact, so a 5 MB JSON that compresses to 500 KB is checked against the compressed size on the broker.

The claim-check pattern avoids the problem: write the large object to object storage, and publish a small record with its location, checksum and metadata. Consumers fetch the object when they need it. The stream stays fast and the object has its own lifecycle.

Where the payload must travel in Kafka, the alternative is chunking. Split it into ordered records with the same key and reassemble in the consumer, with its own failure cases to handle.

The reasoning

`poll()` is where everything happens: joining the group, fetching records, committing offsets with auto-commit, and signalling that the consumer is making progress. If the application does not call `poll()` within `max.poll.interval.ms`, documented at 5 minutes, the consumer is considered stuck and leaves the group.

The heartbeat thread runs on its own. It tells the group coordinator the process is alive, independent of whether `poll()` is being called. If no heartbeat arrives within `session.timeout.ms`, documented at 45 seconds, the coordinator assumes the process died and rebalances.

So a consumer that is alive but slow, say 8 minutes processing one batch, keeps heartbeating and still gets removed, because `max.poll.interval.ms` fired. The fix is to poll more often, process fewer records per poll, or hand work to another thread.

The consumer docs note that with the new `consumer` group protocol, session timeout and heartbeat interval come from broker settings instead of the client.

The reasoning

A brand-new consumer group has no committed offsets. With `latest` it ignores everything already in the topic and reads only what arrives after it joins, which surprises people who expected a backfill. With `earliest` it reads from the oldest retained record, which surprises people who expected to skip the backlog.

It also applies when a committed offset is out of range. That happens when the group was idle long enough for retention to delete the records. It also happens when `offsets.retention.minutes`, documented at 10,080, 7 days, expires the commit itself. Then the consumer resets according to this setting, which can mean silently skipping to the end.

It does not apply when a committed offset exists. A restarted consumer resumes from its commit regardless of this setting. To re-read, you reset the group's offsets with the admin tool or start a new group.

The consumer docs warn that `latest` can skip records when partitions are added mid-run. A new partition has no commit and starts at its end. The docs suggest `by_duration` as a safer default.

The reasoning

With dynamic membership, every restart is two group changes: the old member leaves, triggering a rebalance, and the new member joins, triggering another. A rolling deploy of 12 consumers is 24 rebalances, each pausing consumption.

With a static id, such as the pod name, the coordinator keeps the member's partition assignment through a disconnect. When the restarted process rejoins with the same id, it gets the same partitions and nothing else moves.

The trade is detection time. A static member that dies is only noticed after `session.timeout.ms`, so that value is usually raised for static groups, and a crashed pod holds its partitions idle until then.

Static membership pairs with the cooperative assignor, and with KIP-848's new protocol, to make membership changes incremental rather than stop-the-world.

The reasoning

A fetch request goes to the broker, which replies immediately if it has at least `fetch.min.bytes` available, or waits up to `fetch.max.wait.ms` for more. With the defaults of 1 byte and 500 ms, the broker answers as soon as anything exists, which minimises latency and maximises request count.

Raising `fetch.min.bytes` to, say, 1 MB makes the broker batch replies, cutting request overhead at the cost of up to `fetch.max.wait.ms` of extra latency on a quiet topic. `fetch.max.bytes`, documented at 50 MiB, and `max.partition.fetch.bytes` at 1 MiB cap the reply.

`max.poll.records` does not change the network at all. The consumer buffers what it fetched and hands out at most that many per `poll()`. Lowering it is how you keep processing per poll short enough to stay under `max.poll.interval.ms`.

Throughput comes from fetching big and processing in parallel across partitions. Latency comes from fetching small and often. The settings let you pick a point between.

The reasoning

Confluent's registry applies the configured mode when a producer registers a new schema version. BACKWARD is the default. A change that fails the check is rejected at registration, before any message is produced with it.

BACKWARD protects replay: new readers can process old records. Deleting a field is fine because the new reader does not need it. Adding a field is fine only with a default, because old records lack it.

FORWARD protects slow consumers: old readers can process new records. Adding a field is fine because the old reader ignores it. Deleting a field is fine only if it was optional, because the old reader may expect it.

FULL requires both, which in practice means adding and removing only optional fields with defaults. The transitive forms, such as FULL_TRANSITIVE, compare against every registered version, which is what you want when consumers may be many versions behind.

The reasoning

A source connector produces records and a sink connector consumes them. SMTs sit between the connector and Kafka, applied in a configured order to every record, one at a time, with no state and no access to other records.

Built-in transforms cover the common cases. `InsertField` adds metadata and `ReplaceField` renames or drops. `MaskField` hides values and `TimestampConverter` fixes types. `RegexRouter` picks the topic, `Flatten` unnests, and `ExtractField` lifts a key. Debezium adds its own, such as unwrapping its change envelope.

Because they run per record inside the Connect worker, a heavy transform slows every connector task. Because they are stateless, they cannot deduplicate, join or window. And because they are configuration, they are easy to deploy and hard to test.

Predicates let a transform apply only to records that match a condition, which keeps one connector serving several topics without separate configs.

The reasoning

Kafka Streams runs inside your application; there is no cluster of its own. Each task owns some input partitions and keeps the state for their keys locally, which makes lookups fast and avoids a round trip per record.

Local state is fragile, so the library mirrors it. Each store has a changelog topic named after the application and store, with compaction enabled, so it holds the latest value per key. On failure, the task is reassigned, and the new instance restores the store by reading the changelog from the beginning.

Restoration time is the cost. A large store takes minutes to replay, during which the task is paused. Standby replicas, configured with `num.standby.replicas`, keep a warm copy on another instance so failover is near instant.

KTables are state stores with a changelog; a KStream is stateless. Windowed aggregations keep one store entry per key per window, with retention that drops old windows.

The reasoning

A consumer that throws on a record and restarts reads the same offset again and throws again. The partition is blocked and lag grows while every other partition is fine. Moving the record aside is the only way forward that does not lose it.

In Kafka Connect, sink connectors support this directly: `errors.tolerance=all` with `errors.deadletterqueue.topic.name`, plus context headers that record the failure reason and origin. Source connectors cannot use it, because the failure happens before a Kafka record exists.

In a hand-written consumer, catch the exception, produce the raw record to the DLQ with headers describing the failure, then commit the offset so the partition advances. Retrying a few times first, with backoff, separates transient failures from permanent ones.

A dead-letter topic without an owner is a slow data loss. Alert on its growth and keep its retention long. Build the replay: fix the cause, then re-produce from the DLQ to the main topic or to a retry topic, with the same idempotent sink.

The reasoning

Clusters do not replicate to each other natively. MirrorMaker 2 runs as Connect connectors: a source connector reads the origin cluster, a checkpoint connector mirrors consumer offsets, and a heartbeat connector measures lag. It scales like any Connect deployment.

Replicated topics are renamed, `orders` becomes `east.orders` on the target, so that bidirectional setups do not loop. Record offsets differ between clusters because the target appends in its own order with its own gaps, which is why the checkpoint connector keeps an offset translation map.

Failover uses that map. A consumer group moving to the target cluster looks up where its source offset corresponds and resumes close to where it was, with some duplicates rather than loss.

It is not synchronous replication. Records are copied after they are committed at the source, so a source outage can lose the tail that was not yet mirrored. Stretch clusters and the newer cluster linking features in vendor distributions address the synchronous case differently.

The reasoning

Before KRaft, cluster metadata lived in ZooKeeper and one broker acted as controller by taking a ZooKeeper lock. That meant two systems to operate, a controller failover that re-read all metadata, and a practical ceiling on partition count.

With KRaft, a small set of nodes, typically 3 or 5, run the controller role. They elect a leader by Raft, and that active controller appends every metadata change to the `__cluster_metadata` log. Followers replicate it, and brokers fetch it to keep a local copy, so broker startup and failover are reading a log, not scanning a tree.

Controllers can be dedicated nodes or combined with the broker role on small clusters. The quorum needs a majority to accept writes, so 3 controllers tolerate 1 failure and 5 tolerate 2.

Kafka 3.3 made KRaft production-ready, 3.x supported migrating a ZooKeeper cluster, and 4.0 removed ZooKeeper mode, so a cluster must migrate on a 3.x release before upgrading.

The reasoning

Without tiering, retention is bounded by broker disks, and adding retention means adding brokers. Replaying a year of events meant keeping a year on expensive, replicated disks.

With tiering, a segment is uploaded once it is closed, and the local copy is removed after `local.retention.ms` or `local.retention.bytes`. Consumers reading recent data hit local disk as before. A consumer replaying old offsets is served from the remote tier, slower but complete.

Only the leader uploads, and the remote copy is one copy, so replication factor no longer multiplies the cost of history. Compacted topics are not supported for tiering, since compaction rewrites segments.

It also makes brokers lighter: a broker that fails has less local data to re-replicate, so recovery is faster, which is the operational argument as much as the cost one.

The reasoning

Before headers, metadata had to go in the payload, which coupled every consumer to the envelope format, or in the key, which changed partitioning. Headers give a third place that neither the partitioner nor the deserializer looks at.

Typical uses: a tracing id, so a record can be followed across services. The serialization format or schema reference, so a generic consumer can pick a decoder. A retry counter for retry topics. And Connect's dead-letter context headers, naming the failing connector, the exception and the original offset.

Headers are bytes. The client gives no types, so conventions matter: agree on names and encodings across teams, and keep them small, since they travel with every record and count toward its size.

Brokers do not inspect headers. Filtering on them means fetching the record. Routing by header is a consumer-side or Connect-predicate feature, not a broker one.

The reasoning

A partition is a log with its own offsets. Removing one would mean either deleting its records or appending them to another partition with new offsets, which would invalidate every committed offset and every ordering assumption. Kafka refuses rather than guess.

Adding partitions is a metadata change: new empty logs appear, and new records may land in them. Nothing is rebalanced, so old partitions stay large and new ones start empty, and consumer groups rebalance to pick up the new partitions.

The ordering cost is the real one. `hash(key) % 12` and `hash(key) % 48` differ for most keys. Events for `order_42` that were on partition 3 now go to partition 27. A consumer may see the new partition's events before it finishes the old one's.

The practical rule is to size partitions for the throughput you expect in two years, with headroom. Treat a change in count as a migration, with a quiet window and consumer coordination.

The reasoning

Kafka sits between the systems that produce data and the systems that use it. Producers write once; any number of consumer groups read the same records, each keeping its own position, and a slow consumer does not slow a fast one.

Durability and retention make it a buffer: a burst is absorbed, a downstream outage is survived, and a bug is fixed by replaying. That decoupling is the main reason to put it between a service and a warehouse instead of writing directly.

It is not a database. There are no secondary indexes, no updates in place, no queries beyond reading forward from an offset. Compacted topics give a latest-value-per-key view, but anything else is built on top in Kafka Streams, ksqlDB or a downstream store.

In a data platform the common shapes are few. Services to Kafka to a lakehouse, through Connect or Spark. Databases to Kafka through Debezium. Kafka to Flink or Streams for real-time aggregates. Kafka to a search index or a cache for serving.

The reasoning

Each broker has an id and owns a set of partition replicas. For the ones it leads, it accepts writes and serves reads. For the ones it follows, it fetches from the leader. A single broker can hold thousands of partitions.

Clients connect to any broker to fetch metadata, learn which broker leads each partition, and then talk to leaders directly. There is no router in the path, which is why throughput scales with brokers.

A cluster's capacity is the sum of its brokers' disks and network, divided across partitions. Adding a broker adds capacity only once partitions are reassigned to it, which is an explicit operation.

With KRaft, some nodes also run the controller role that holds cluster metadata; on small clusters those are the same processes as the brokers.

The reasoning

Having one leader per partition is what keeps writes ordered: every append goes through one broker, which assigns the next offset. Followers replicate that log exactly.

The controller watches broker liveness. When a broker drops, every partition it led needs a new leader. The controller chooses from the ISR, prefers the replica listed first in the assignment when it is in sync, and writes the change to the metadata log.

Clients see a `NotLeaderOrFollower` error on their next request, refresh metadata, and retry against the new leader. With `acks=all` and `min.insync.replicas` of 2, nothing acknowledged is lost in the handover.

If no replica is in sync, the partition is unavailable until an ISR member returns. An unclean election, off by default, would promote a stale follower instead, and lose the records it never got.

The reasoning

Delete retention suits event streams: clicks, orders, metrics. The data is needed for a window and then not at all, and the window sets the disk cost.

Compaction suits changelogs: a user's current profile, a product's current price, a Streams state store's backup. A consumer reading the whole topic gets the latest value per key and nothing stale, which is how a KTable or a Connect offset topic is rebuilt.

Compaction runs in the background on closed segments, keeping the head of the log uncompacted for a while. A record with a null value is a tombstone: it deletes the key after `delete.retention.ms`, long enough for consumers to see the deletion.

A topic can use both policies together, compacting old data and still deleting it past a time limit, which bounds a changelog that keeps acquiring new keys.

The reasoning

The storage model is the difference. Kafka appends to a partitioned log and keeps it; a consumer's position is a number it stores. RabbitMQ holds messages until delivered and acknowledged, then forgets them, with per-message routing through exchanges and bindings.

That gives Kafka replay, multiple independent readers, and ordering per partition. The cost is per-message features: no per-message acknowledgement to the broker, no priority, no routing beyond topic and partition. It gives RabbitMQ flexible routing, dead-lettering and work-queue semantics, at the cost of replay and of fan-out to many readers.

Scale differs too. Kafka partitions scale consumers by partition count and handle millions of messages per second on commodity disks. Queues scale by adding consumers to one queue and are usually bounded by memory and acknowledgement rate.

Data pipelines are streams with several consumers and a need to replay, which is why Kafka is the default there. Background jobs, retries with per-task acknowledgement, and request-reply patterns are queue work.

The reasoning

The model is the same: a partition key hashes a record to a shard or partition, ordering holds within it, consumers track positions. Someone who knows one can read the other's docs in an afternoon.

Kinesis removes operations. No brokers to size, resharding by API, on-demand mode that scales with traffic, and a retention knob instead of disks. Its limits are per shard, and enhanced fan-out is needed when several consumers read at full speed.

Kafka gives control and portability: no per-shard throughput caps, compaction, transactions, exactly-once between topics, Kafka Streams, and the Connect ecosystem. It costs an operator or a managed fee, and it ties ordering to a partition count you chose.

Choose Kinesis inside AWS when the workload fits its limits and the team is small. Choose Kafka when you need its ecosystem, run outside one cloud, or need the semantics only Kafka has.

The reasoning

The producer batches in the background, so `send()` cannot know the outcome. It returns immediately. The outcome arrives later, through the future or the callback, with either the record's partition and offset or an exception.

A callback that logs and counts failures is the minimum. A callback that writes the failed record to a local failure store is what an ingestion layer needs. A producer that drops records on the floor is at-most-once without saying so.

Retries happen before the callback fires: the producer retries transient errors on its own, up to `delivery.timeout.ms`, and only then reports failure. The idempotent producer, documented as enabled by default, makes those retries safe from duplicates.

Calling `.get()` on every future makes sending synchronous and throughput collapses. The callback keeps the pipeline asynchronous while still observing every result.

The reasoning

Connect is a framework of workers. A connector is a JSON configuration naming the plugin, the topics, the external endpoint and the number of tasks. Workers split the connector's work into tasks and run them, rebalancing when a worker dies.

Source connectors own the hard part of ingestion. They poll or tail the source. They track their own offsets in Connect's internal topic, so a restart resumes. And they convert records to Kafka's format with a converter such as Avro or JSON.

Sink connectors own delivery: consuming with a group, batching writes to the target, committing offsets after a successful write, and routing failures to a dead-letter topic when configured.

Connect gives you offsets, retries, scaling and a REST API for free. The price is that complex logic does not fit; it is for moving data, with light shaping through single message transforms.

The reasoning

Replication is pull. Followers fetch from the leader in a loop, and the leader learns how far each follower has got from the offsets they ask for. A follower that falls more than `replica.lag.time.max.ms`, documented at 30 seconds, behind is dropped from the ISR.

The leader advances the high watermark to the minimum of the ISR members' log end offsets. Records above it exist on the leader but are not yet safe, because a leader failure could lose them if a follower without them is elected.

Consumers fetch up to the high watermark only. That is the visibility rule that makes `acks=all` meaningful. An acknowledged record is below the watermark, so it survives a failover. No consumer ever sees a record that later disappears.

When a follower rejoins after being out of sync, it truncates its log to the leader's state and catches up. Only then is it readmitted to the ISR. That is why a long-lagging replica can take minutes to count again.

The reasoning

The commit is what a restart resumes from. It is stored on the brokers, in a compacted topic replicated `offsets.topic.replication.factor` times, documented at 3, so it survives broker failure and lets any instance of the group pick up.

Committing the next offset rather than the last one means a consumer that restarts reads exactly the record after the one it finished. Off-by-one here is a duplicate or a skipped record.

Auto-commit is at-least-once at best: it commits what the last poll returned, whether or not processing finished, and only when `poll()` is next called. A crash between poll and commit replays the batch; a crash after commit but before the work completed loses records.

Manual commit, with `commitSync()` after processing or `commitAsync()` with a callback, gives control. Committing offsets together with the output, in the sink's own transaction, is how a consumer gets effectively-once.