Apache Kafka Interview Questions for Data Engineers

01

The 2 kinds of Apache Kafka question

Two kinds: explain how it works, then use it on a stated problem. Each card gives the answer to say out loud, the answer most people give, and where the interviewer goes next.

• 25ConceptualThe log underneath everything. What a partition promises and what it does not, what happens to a consumer group when one member dies, where an offset is stored, and what acks=all actually waits for.Topics, partitions & keys · Consumer groups & rebalancing · Offsets & commit strategies · Brokers, replication & ISR→
• 25Scenarios & DesignConsumer lag is climbing, one partition is hot, a rebalance produced duplicates, a bad message is blocking a partition. Diagnose it from the symptoms, then design the topic, the keys and the delivery guarantee for a stated load.Consumer lag & throughput tuning · Delivery semantics · Dead-letter topics & poison messages · Idempotent producers & transactions→

02

The ones that keep coming up

Answered here, in short. The full reasoning, the wrong answer and the follow-up are one click away.

1. 01Explain Apache Kafka's architecture: brokers, topics, partitions, producers, consumers and the controller.Producers append records to topics; each topic is split into partitions, which are replicated logs spread across brokers, with one leader per partition serving reads and writes. Consumers pull from partitions and track their position as offsets, coordinated in consumer groups, and a KRaft controller quorum holds the cluster metadata — ZooKeeper is gone as of Kafka 4.0.→
2. 02What is a Kafka consumer, and what is a consumer group?A consumer is a client that pulls records from partitions and tracks its offset in each. A consumer group is a set of consumers sharing one group.id that divide a topic's partitions between them — each partition to one member — so the group processes the topic once, in parallel, while other groups read the same data independently.→
3. 03What is ISR (in-sync replicas) in Kafka, and how is it different from the replication factor?The replication factor is how many copies of a partition exist; the ISR is the subset — leader included — that is currently caught up with the leader. Kafka acknowledges acks=all writes only once every ISR member has them and, by default, elects new leaders only from the ISR, so it is the ISR that says how many real copies you have right now.→
4. 04What is log compaction in Kafka, and when would you use it instead of time-based retention?Compaction (cleanup.policy=compact) keeps at least the latest record for every key and removes older records for that key in the background, so the topic becomes a changelog of current state; a record with a null value, a tombstone, deletes the key. Use it when consumers need the latest value per key rather than the full history — CDC tables, Kafka Streams changelogs, and __consumer_offsets itself.→
5. 05What is the role of ZooKeeper in Kafka, and can Kafka run without it?Until Kafka 4.0, ZooKeeper held the cluster metadata — broker membership, topic and partition configuration, ISR state — and elected the controller broker that managed leadership. Kafka has run without it in KRaft mode since 3.3, where a Raft quorum of controllers keeps metadata in an internal log, and 4.0 removed ZooKeeper mode entirely.→
6. 06What do acks=0, acks=1 and acks=all mean for a Kafka producer?acks=0: the producer does not wait for any response, so it never learns about failures. acks=1: the leader has written the record to its log; acks=all, the default since Kafka 3.0: every in-sync replica has it — safe against one broker loss only when min.insync.replicas is at least 2.→
7. 07Explain at-most-once, at-least-once and exactly-once delivery in Kafka.At-most-once: commit the offset before processing, so a crash loses records but never repeats them. At-least-once: process, then commit, so a crash repeats records but never loses them; exactly-once in Kafka means a read-process-write loop whose output and input offsets commit in one transaction, and it covers only results written back to Kafka.→
8. 08What is consumer lag in Kafka, and how do you monitor it?Lag is, per partition, the log-end offset minus the group's committed offset — how many records the group has not yet processed. Watch it per partition and as time, with kafka-consumer-groups.sh --describe, the consumer's records-lag-max metric, or an exporter such as LinkedIn's Burrow, and alert on a trend rather than a fixed number.→
9. 09What is Kafka Connect, and when would you use it instead of writing your own producer or consumer?Kafka Connect is Kafka's framework for moving data between Kafka and other systems through configured connectors rather than code: source connectors bring data in, sink connectors write it out, and the framework handles scaling, offsets, retries and restarts. Use it when a solid connector exists — Debezium for databases, S3, JDBC — and write your own client when the logic is specific to your domain.→
10. 10What is the difference between a KStream and a KTable in Kafka Streams?A KStream is an unbounded sequence of independent events — two records with the same key are two things that happened. A KTable is a changelog view in which each record is the latest value for its key, replacing the previous one, with a null value deleting it — so aggregating a stream produces a table, and a table's changes can be read as a stream.→

03

What Apache Kafka interviews actually cover

• 01Topics, partitions & keys
• 02Brokers, replication & ISR
• 03Producer acks, retries & batching
• 04Idempotent producers & transactions
• 05Consumer groups & rebalancing
• 06Offsets & commit strategies
• 07Delivery semantics
• 08Ordering guarantees
• 09Retention & log compaction
• 10Schema Registry & serialization
• 11Kafka Connect & CDC
• 12Kafka Streams & stream processing
• 13Consumer lag & throughput tuning
• 14KRaft & ZooKeeper
• 15Dead-letter topics & poison messages

04

Learn it, then practise it