What You'll Master Here
producers append events, the log keeps them durable and ordered, consumers read forward at their own pace.
This chapter opens the streaming half of the course. Everything so far could be done in batch; from here on the data never ends. Streaming architecture is how you move and process an endless flow of events the instant they happen, and it has its own vocabulary, its own components, and its own ways of failing.
The good news is that the whole architecture rests on one beautifully simple idea: the log. A streaming platform like Kafka is, at heart, an append-only log that producers write to and consumers read from. Once you understand the log, topics, partitions, offsets, and consumer groups all fall out of it naturally.
By the end you will be able to draw the producer → broker → consumer architecture, explain why partitions are the key to both scale and ordering, and reason about offsets and consumer groups, the machinery LinkedIn invented Kafka to provide and that now moves trillions of messages a day across the industry.
A streaming platform is an append-only log in the middle: producers append events, the log keeps them durable and ordered, consumers read forward at their own pace.
Streaming is how real-time decisions get made: fraud blocks, surge pricing, live recommendations. The log-based architecture in this chapter is the foundation under all of it, and it is the model nearly every streaming system (Kafka, Kinesis, Pub/Sub) shares.
- event
- An immutable record that something happened (a click, a payment) at a point in time.
- broker / log
- The durable, append-only store in the middle that holds events between producers and consumers.
- producer
- A program that appends events to the log.
- consumer
- A program that reads events from the log and acts on them.
Picturing streaming as "a fast database" rather than a log. You misunderstand ordering, replay, and consumer offsets, which only make sense in the log model.
Think in terms of an append-only log, not a queue you drain.
Separate producers, the broker, and consumers in your mental model.
Remember events are immutable facts, never updated in place.
A stream is a log: an ordered, append-only sequence of immutable events. Producers write to the end, consumers read forward and remember their position. Everything else is detail on top of this.
Start with the log and the three roles, then partitions (the heart of scale and order), then offsets and consumer groups. Each builds on the last.
Streaming architecture is an append-only log with producers, a broker, and consumers; master the log and topics, partitions, offsets, and consumer groups all follow.
- Define producer, broker, and consumer in one sentence each.
- Explain why an event is immutable.