Streaming Architecture Fundamentals

In the last chapter, the shop sent each order as an event. Something must carry those events from the shop to the code that uses them. This chapter shows how that works.

What you already know

• A stream is a feed that sends each event as it happens.
• Event time is when the order was placed. Processing time is when your pipeline sees it.
• A late event arrives after events that happened later than it. A window is a block of time. A watermark decides when a window is done.
• At-least-once means an event can arrive twice. A MERGE on order_id makes a repeat safe.
• Kafka stores events in order. Lag is how far a reader is behind.

What you will learn

1. The parts of a streaming system, and why it keeps a log.
2. How long the log keeps events, and how replay works.
3. Partitions, and why order exists only inside one partition.
4. Offsets and consumer groups.
5. What a stream processor and a sink do.
6. Consumer lag and backpressure.
7. The producer and consumer settings that decide if data is lost.

We use the same shop and the same made-up day, 2026-01-22. Orders 110 to 114 are the same. Orders 150 and 151 are new.

This table lists the seven orders in the order they arrive:

A streaming system has a few parts. Each part has one job. This table names them, with the shop as the example:

The broker keeps a log. A log is a list where events are only added at the end. The producer appends, and nobody changes an old entry.

Each entry gets an offset. An offset is its position number in the log, starting at 0. This is the topic orders with one log:

-- A topic is a log. New events are added at the end.
-- Offset = position in the log, starting at 0.
-- Producer appends:  order 110, 111, 112, 113, 114, 150, 151

Reading does not remove an event. This is the big difference from a queue, where a message is gone once someone takes it.

Because of this, many readers can use the same topic. Each reader remembers its own position. This table shows two readers at one moment:

The loader and the counter do not disturb each other. A new reader added later can start at offset 0, if the broker still keeps it.

Kafka calls one event in the log a record. Amazon Kinesis, a similar service, calls a topic a stream and an offset a sequence number. The idea is the same.

The log cannot keep events for ever. Retention is the rule for how long the broker keeps them. A broker deletes old events when they pass the time limit, or when the log grows past a size limit you set.

In Kafka, the broker setting log.retention.hours has a default of 168 hours. That is 7 days. You can set a different limit for each topic.

Replay means reading old events again. You move a reader back to an earlier offset.

Say a bug in the counter gave wrong counts since order 113. This table shows the fix:

Replay only works for events the broker still keeps. If offset 3 was already deleted, you cannot go back to it.

A new reader group can also start from the oldest kept event. In Kafka, the consumer setting auto.offset.reset=earliest does this. The default, latest, skips everything already in the topic.

One log on one machine can only go so fast. So a topic is split into partitions. A partition is one ordered log, and different partitions can live on different machines.

The producer picks a partition for each event. It uses the key, a value you send with the event, such as customer_id. The same key goes to the same partition, as long as the number of partitions does not change.

Kafka hashes the key: it turns the key into a number. It uses that number to pick the partition. If you add partitions later, new events for some keys go to a different partition.

To keep the example easy to check, we use a made-up rule: partition = customer_id mod 2 (the remainder after dividing by 2). Customers 2 and 4 go to p0. Customers 1, 3 and 5 go to p1.

-- Made-up rule, so you can check every row:
--   partition = customer_id mod 2
-- Real Kafka: partition = hash(key) mod number_of_partitions

Look at customer 2. Orders 110 and 150 are both in p0, at offsets 0 and 2. A reader sees 110 before 150, always.

Now look across partitions. Order 110 is in p0 and order 111 is in p1. Kafka makes no promise about which one a reader sees first.

So order exists only inside one partition. If events must stay in order, give them the same key. Kinesis calls a partition a shard.

A consumer must remember how far it has read. It saves its position as a committed offset. In Kafka, the committed offset is the offset of the next record to read.

A consumer group is a team of consumers that share one topic. Kafka gives each partition to exactly one consumer in the group. Each group keeps its own committed offsets.

This table shows two groups on the topic orders at one moment:

The loader has two consumers, so each reads one partition. The counter has one consumer, so it reads both. The two groups do not disturb each other.

Say consumer B stops. Kafka moves p1 to consumer A, a move called a rebalance.

A starts p1 at its committed offset, 2, so the next order it reads is 113.

Now the opposite case. Say the loader had three consumers. A third consumer would sit idle, because there are only two partitions.

The number of partitions limits how many consumers can work.

Now see why the commit moment matters. This loop reads partition p1 and writes each order to the warehouse:

for record in consumer:        # pseudocode: reads p1 in offset order
    write_to_warehouse(record)    # MERGE on order_id
    consumer.commit(record)       # save "next offset to read"

Commit after the write gives at-least-once: you may read a record twice, but you never lose it. If you commit first and then write, a crash in between loses the record. The chapter Delivery Guarantees & Ordering covers this in depth.

A stream processor is a consumer that does work on the events. A sink is where it writes the result. The chapter Stream Processing teaches this work in detail.

Here the counter is the stream processor. It reads both partitions, counts orders in five-minute windows by event time, and writes to a sink. The sink is a table called orders_per_window.

This table shows the sink after the counter has read all seven orders. The watermark waits 3 minutes:

The newest event time is 10:10, so the watermark is 10:07. The first window ended at 10:05, so it is closed. The second window ends at 10:10, so it is still open.

The processor must remember the counts between events. That memory is called state.

If the processor restarts, it needs its state back. It loads a saved copy and replays the log from the offsets saved with it.

Consumer lag is how many records a consumer has not read yet. Lag in one partition is the log end offset minus the committed offset. The log end offset is the next offset to write.

p0 has 3 records, so its log end offset is 3. p1 has 4 records, so its log end offset is 4. This table gives the lag of each group:

The loader has total lag 2: both waiting orders are in p1. The counter has total lag 3.

A lag that stays near zero is healthy. A lag that keeps growing is a warning.

Backpressure means a slow stage makes the stage before it slow down. Kafka works differently.

Consumers pull records at their own speed, so a slow consumer does not slow the producers. The log absorbs the difference, and lag grows.

Here is a made-up case. Orders arrive at 100 per second, and the loader handles 80 per second. Lag grows by 20 per second, which is 1,200 per minute.

A producer gets a "success" answer when the write is done. The acks setting decides what the producer waits for first.

The broker that handles a partition is the leader. Other brokers keep copies, called followers.

This table shows the three values of acks:

An idempotent producer sends each record with a sequence number. The broker then skips a record that a retry sent twice. Without it, a retry can store the same order two times.

This works only inside one running producer. If your app restarts and sends again, you still need a MERGE.

Defaults depend on the client. The Java client from Kafka 3.0 on uses acks=all and an idempotent producer. Older Java clients use acks=1, and other clients, such as the Python ones, have their own defaults.

Check the version you run, and set both settings on purpose.

On the consumer side, one setting matters most. Auto-commit saves the offset on a timer. It can save the offset of a record you have not finished.

Turn it off and commit after the write, as in the loop earlier. These are the safe settings, with their Kafka names:

• acks=all and enable.idempotence=true on the producer.
• enable.auto.commit=false, and commit after the write, on the consumer.
• A longer max.poll.interval.ms than your slowest batch. If a batch takes longer, Kafka treats the consumer as dead and rebalances.
• auto.offset.reset=earliest only for a new group that must read old events.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.