Streaming Architecture Fundamentals

This chapter opens the streaming half of the course. Everything so far could be done in batch; from here on the data never ends. Streaming architecture is how you move and process an endless flow of events the instant they happen, and it has its own vocabulary, its own components, and its own ways of failing.

The good news is that the whole architecture rests on one beautifully simple idea: the log. A streaming platform like Kafka is, at heart, an append-only log that producers write to and consumers read from. Once you understand the log, topics, partitions, offsets, and consumer groups all fall out of it naturally.

By the end you will be able to draw the producer → broker → consumer architecture, explain why partitions are the key to both scale and ordering, and reason about offsets and consumer groups, the machinery LinkedIn invented Kafka to provide and that now moves trillions of messages a day across the industry.

event

An immutable record that something happened (a click, a payment) at a point in time.

broker / log

The durable, append-only store in the middle that holds events between producers and consumers.

producer

A program that appends events to the log.

consumer

A program that reads events from the log and acts on them.

At the centre of every streaming platform is a log: an ordered, append-only sequence of events. Producers append new events to the end; the broker stores them durably; consumers read forward from wherever they left off. Crucially, reading does not delete, unlike a traditional queue, the log retains events so many consumers can read the same stream independently, and any of them can rewind and replay.

This design is what makes streaming so powerful. One producer (the orders service) can feed many consumers at once: a fraud detector, a warehouse loader, a real-time dashboard, all reading the same log at their own pace. Add a new consumer next year and it can replay history from the start. The log decouples who produces data from who uses it.

A topic is just a named log for one kind of event (an "orders" topic, a "clicks" topic). Producers write to a topic; consumers subscribe to it. This is the producer → broker(topic) → consumer picture, the irreducible core of Kafka, Kinesis, and Pub/Sub alike.

log (append-only)

An ordered sequence where events are only ever added to the end, never changed.

topic

A named log for one category of events (orders, clicks, payments).

retention

How long the broker keeps events; reads do not delete them.

replay

Re-reading the log from an earlier position, e.g. to reprocess with new logic.

A single log on one machine would cap your throughput and storage. So a topic is split into partitions, each an independent ordered log, and they can live on different machines. Partitions are the single most important concept in streaming, because they provide two things at once: parallelism and ordering.

Parallelism: with three partitions, three consumers can read in parallel, tripling throughput. Ordering: events are ordered within a partition, but not across partitions. This is the key trade-off. If you need all of one customer’s events in order, you must send them to the same partition, usually by hashing a key (customer_id) so the same key always lands on the same partition.

The example shows the consequence. Events with key "u1" all go to partition 0 and stay perfectly ordered; "u2" goes to partition 1. Across partitions there is no global order, partition 0 and partition 1 interleave however the consumers happen to read them. Choosing the partition key is therefore choosing your ordering guarantee.

partition

One independent ordered log within a topic; the unit of parallelism and ordering.

partition key

The value hashed to choose a partition; same key → same partition → ordered together.

per-partition ordering

Events are ordered within a partition but not across partitions.

hot partition

A partition receiving disproportionate traffic (a skewed key), bottlenecking the topic.

-- Producer hashes the key to pick a partition:
--   partition = hash(key) % num_partitions
-- Same key -> same partition -> guaranteed order for that key.

Because the log is not consumed by reading, each consumer must remember how far it has read. That bookmark is the offset: a per-partition position. A consumer commits its offset as it processes, so after a restart it resumes from there rather than re-reading everything or skipping ahead. Offsets are why consumers can crash and recover cleanly.

To scale consumption, related consumers form a consumer group. The group shares the work: each partition is assigned to exactly one consumer in the group, so with three partitions and three consumers, each handles one. Add a fourth consumer and it sits idle (no partition to take); drop to two and one consumer takes two partitions. This automatic rebalancing is how streaming consumers scale horizontally.

Two different applications (a fraud service and an analytics loader) use two different consumer groups, and each group independently tracks its own offsets on the same topic. This is the final piece: one durable log, partitioned for scale and order, read by multiple groups that each remember their own position. That is the whole architecture.

offset

A consumer’s position in a partition; committed so it can resume after a restart.

consumer group

A set of consumers that share a topic’s partitions, one partition per consumer.

rebalancing

Reassigning partitions among group members when consumers join or leave.

offset commit timing

When a consumer records progress; commit-before-process risks loss, commit-after risks duplicates.

# SAFE DEFAULT — commit AFTER processing → at-least-once.
# A crash before commit just reprocesses; idempotency makes that harmless.
for event in consumer:
    process(event)            # do the work first
    consumer.commit(event)    # then record progress

# DANGEROUS — commit BEFORE processing → at-most-once.
for event in consumer:
    consumer.commit(event)    # progress recorded...
    process(event)            # ...crash here = event is GONE forever

# Same loop, two lines swapped — opposite correctness guarantee.

Every streaming tutorial shows three lines that send a message. None of them shows the settings that decide whether that message still exists after a broker restarts.

The dial below is the one that matters most: what your producer waited for before it told you the write succeeded. Then the consumer side, and the same thing on Kinesis.

in-sync replica (ISR)

A follower that has caught up with the leader. acks=all waits for all of them, and min.insync.replicas sets how many must exist for a write to be allowed at all.

idempotent producer

The broker deduplicates producer retries using a sequence number, so a retried send cannot append a second copy.

consumer lag

How many records sit between a consumer’s committed offset and the end of the partition. The single most useful streaming metric.

schema registry

A service holding the schema for each topic and refusing producer changes that would break existing readers.

Marlow checkout → KafkaYour producer returned success. Which of these did that actually mean?acks=1Reset

checkoutPRODUCEREmits one order event.→leaderBROKER 1Writes to its own log.→followerBROKER 2Replicates from the leader.→consumerREADSOnly sees committed records.

acks=0acks=1acks=all

The leader has itthroughput — High

The leader wrote the record to its own log and replied. Followers may not have it yet.

survives	loses
A producer crash, a consumer crash, an ordinary retry.	Records acknowledged by a leader that then dies before any follower replicates them.

Acknowledged does not mean durableData can be acknowledged and then vanish, with no error anywhere. Fine for metrics you can lose; wrong for an order.

The producer, configured

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": BROKERS,
    "acks": "1",
    "linger.ms": 20,                # batch for 20ms — throughput, at 20ms of latency
    "compression.type": "zstd",
})

def on_delivery(err, msg):
    if err:                          # this is the only place you learn it failed
        log.error("delivery failed", key=msg.key(), error=err)

producer.produce(
    topic="orders",
    key=order["order_id"],          # same key → same partition → ordered
    value=json.dumps(order).encode(),
    on_delivery=on_delivery,
)
producer.flush()                     # block until the queue drains before exit

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": BROKERS,
    "group.id": "marlow-orders-loader",
    "enable.auto.commit": False,        # we decide when a record is "done"
    "auto.offset.reset": "earliest",    # a brand-new group reads from the start
    "max.poll.interval.ms": 300000,     # longer than your slowest batch, or you
                                        # get kicked out mid-work and rebalance
    "session.timeout.ms": 45000,
})
consumer.subscribe(["orders"])

try:
    while True:
        msgs = consumer.consume(num_messages=500, timeout=1.0)
        if not msgs:
            continue

        batch = [json.loads(m.value()) for m in msgs if not m.error()]
        write_batch_idempotently(batch)   # keyed on order_id — see Chapter 11

        consumer.commit(asynchronous=False)   # only now is it acknowledged
finally:
    consumer.close()                          # leaves the group cleanly,
                                              # so a rebalance happens at once

import boto3

kinesis = boto3.client("kinesis")

# --- producing: PutRecords batches up to 500 records per call ---------------
kinesis.put_records(
    StreamName="marlow-orders",
    Records=[{
        "Data": json.dumps(o).encode(),
        "PartitionKey": o["order_id"],      # same role as a Kafka message key
    } for o in orders],
)
# NOTE: put_records is partially fallible — check the response!
#   failed = [r for r in resp["Records"] if r.get("ErrorCode")]
# and retry only those, or you will silently drop records.

# --- consuming: iterate a shard ---------------------------------------------
it = kinesis.get_shard_iterator(
    StreamName="marlow-orders", ShardId="shardId-000000000000",
    ShardIteratorType="AFTER_SEQUENCE_NUMBER",
    StartingSequenceNumber=load_checkpoint(),
)["ShardIterator"]

while it:
    resp = kinesis.get_records(ShardIterator=it, Limit=1000)
    handle(resp["Records"])
    save_checkpoint(resp["Records"][-1]["SequenceNumber"])  # after handling
    it = resp["NextShardIterator"]

import requests, json, time
from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": BROKERS, "acks": "all",
                     "enable.idempotence": True})
session = requests.Session()
cursor = load_cursor("payments")

while True:
    r = session.get(f"{API}/payments", timeout=(5, 30),
                    params={"updated_since": cursor, "limit": 500})
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", 30)))
        continue
    r.raise_for_status()
    page = r.json()

    for p in page["data"]:
        producer.produce("payments", key=p["payment_id"],
                         value=json.dumps(p).encode())
    producer.flush()                    # publish BEFORE moving the cursor
    save_cursor("payments", page["next_cursor"])

    if not page["next_cursor"]:
        time.sleep(15)                  # caught up: poll gently

from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema = """
{ "type": "record", "name": "Order",
  "fields": [
    {"name": "order_id",  "type": "string"},
    {"name": "total_gbp", "type": {"type": "bytes", "logicalType": "decimal",
                                   "precision": 12, "scale": 2}},
    {"name": "region",    "type": ["null", "string"], "default": null}
  ]}
"""

serializer = AvroSerializer(SchemaRegistryClient({"url": SR_URL}), schema)

# BACKWARD compatibility (the usual default) permits: adding an optional field,
# removing a field with a default. It refuses: renaming, retyping, or adding a
# required field — the registry rejects the producer at startup, not at 3 a.m.

Five scenarios built on the log: partitions, consumer groups, and the two things that break both — a hot key and consumers who want different answers.

Build them on the pipeline canvas with the chapter closed. Each is graded against the scenario’s own requirement rules rather than against a model answer, so there is more than one design that passes — and a design that does not pass is told exactly which requirement it missed.