Lambda, Kappa & Streaming-First

This chapter zooms out from individual streaming mechanics to whole-system architecture. The central question is timeless: you want data that is both fresh (streaming) and complete/accurate (batch), so how do you combine them? The two famous answers are the Lambda and Kappa architectures, and the modern streaming-first view.

Lambda runs batch and streaming side by side and merges their results. Kappa argues that is needless duplication and keeps a single streaming path, reprocessing by replaying the log. The debate between them is really a debate about complexity versus capability, and it teaches you how to think about freshness and reprocessing at the architecture level.

By the end you will be able to draw both architectures, explain the "two codebases" problem that motivated Kappa, reason about how reprocessing works in each, and make a sober recommendation, which, as usual, favours the simplest design that meets the requirement.

Lambda architecture

A design with a batch layer (accurate) and a speed layer (fast), merged into one serving view.

Kappa architecture

A streaming-only design that reprocesses history by replaying the log through the same code.

reprocessing

Recomputing results after a logic change or bug, the operation that distinguishes the two.

serving layer

Where query-ready results live for consumers, fed by the architecture above it.

The Lambda architecture, named by Nathan Marz, runs two parallel pipelines over the same incoming data. The batch layer reprocesses all the data periodically to produce accurate, complete results, slow but correct. The speed layer processes the same data as a stream to produce fast, approximate, up-to-the-second results. A serving layer merges them: recent data comes from the speed layer, older data from the batch layer.

The appeal is that you get the best of both: real-time freshness from the speed layer and eventually-corrected accuracy from the batch layer, which overwrites the approximate results as it catches up. For years this was the standard way to build systems that needed both, and many large platforms ran it.

The cost is the defining problem of Lambda: two codebases. The same business logic (say, "count active users") must be implemented twice, once in batch (Spark/SQL) and once in streaming (Flink/Kafka Streams), and kept perfectly in sync. Any divergence means the two layers disagree, and maintaining two implementations of every metric is a heavy, bug-prone burden.

batch layer

The accurate, complete pipeline that periodically reprocesses all data.

speed layer

The real-time streaming pipeline producing fast, approximate results.

serving layer

Where batch and speed results are merged for queries.

two-codebase problem

Maintaining the same logic twice (batch and streaming), kept in sync, Lambda’s core drawback.

The Kappa architecture, proposed by Jay Kreps (a Kafka creator), makes a bold simplification: delete the batch layer. Keep only the streaming pipeline. There is one codebase, one engine, one set of logic. This removes the two-codebase problem entirely.

The obvious objection is "but how do you reprocess history without a batch layer?". Kappa’s answer is elegant: reprocessing is just replaying the log. Because the streaming platform retains events (Chapter 13), you can rewind a consumer to the beginning and re-run the same streaming code over all of history, producing corrected results, then cut over to the new output. The stream code is the only code; batch is just streaming over old data.

Kappa became practical because stream processors grew powerful enough (Flink especially) to handle both real-time and large reprocessing workloads with one engine. It is simpler and increasingly the default for new systems, but it assumes your streaming engine and log retention can handle reprocessing the full history, which is not always true for enormous or long-lived datasets.

streaming-only

A single streaming pipeline with no separate batch layer.

replay

Rewinding the log and re-running the streaming code over historical events to reprocess.

cutover

Switching consumers to the reprocessed output once a replay completes.

retention requirement

The need to keep enough log history to replay, a key Kappa assumption.

As with batch versus streaming, the senior move is to resist the most impressive option and choose the simplest one that meets the requirement. Most workloads do not need real-time at all and should be plain batch (Chapters 2–12). Among those that do, Kappa’s single path is usually preferable to Lambda’s duplicated one, unless a specific constraint forces the split.

When might Lambda still make sense? When the batch and speed layers genuinely need different engines, for example, a heavy historical recomputation that is impractical to run as a replay, alongside a lightweight real-time approximation. And many real systems are neither pure Lambda nor pure Kappa: they are mostly batch with a few streaming paths, chosen per use case, which is the pragmatic reality the comparison below captures.

The unifying principle of this whole course applies one last time at the architecture level: complexity is a cost you pay forever. Two pipelines, two codebases, and a merge layer are justified only by a requirement that one path cannot meet. Start from the requirement, prefer one path, and add the second only when forced.

streaming-first

Defaulting to a single streaming path (Kappa-style) for real-time needs.

hybrid platform

A mostly-batch platform with selected streaming pipelines per use case.

requirement-driven design

Choosing architecture from the freshness/accuracy requirement, not from fashion.

cost of complexity

The permanent maintenance burden each extra pipeline or codebase adds.

Lambda and Kappa are the two names in the literature. The thing most companies actually run is a third shape that neither paper describes — streaming into a lakehouse table that batch also reads.

Below: all three drawn with real components, the reprocessing story for each, and the code that makes the third one work.

speed layer

The streaming half of a Lambda architecture — fast, approximate, and overwritten later by the batch result.

log replay

Rebuilding a result by reading a topic from the beginning. It only works while the log still retains that history.

streaming lakehouse

A continuously written table format that batch and interactive queries also read, so one table is both the stream’s output and the warehouse’s input.

reconciliation query

The scheduled comparison proving two layers still agree. Lambda needs one; the other two shapes do not.

# 1. start a parallel job over the SAME topic, from the beginning
(spark.readStream.format("kafka")
    .option("subscribe", "orders")
    .option("startingOffsets", "earliest")     # the whole retained history
    .load()
    .transform(revenue_logic)                  # the FIXED logic, one copy of it
    .writeStream
    .option("checkpointLocation", "s3://ckpt/live-revenue-v2/")   # a NEW checkpoint
    .toTable("gold.live_revenue_v2"))

# 2. wait for v2 to catch up with v1 (compare max event_time, or lag = 0)
# 3. repoint readers at v2, then retire v1 and its checkpoint

# The constraint that makes or breaks this: retention.
#   retention.ms must cover the history you may ever need to replay,
#   or "replay from the beginning" quietly means "from seven days ago".

# the stream writes into a table format, not a bespoke serving store
(orders_stream
    .writeStream
    .format("delta")                       # or iceberg
    .outputMode("append")
    .option("checkpointLocation", "s3://ckpt/orders/")
    .option("mergeSchema", "true")
    .trigger(processingTime="1 minute")
    .toTable("silver.orders"))

-- and every consumer just reads a table, whenever it likes
SELECT region, SUM(total_gbp)
FROM   silver.orders
WHERE  dt = CURRENT_DATE()
GROUP  BY region;

-- a batch backfill writes to the SAME table, with the same logic,
-- because the table is the interface rather than the pipeline.

-- the serving view: speed layer for today, batch for everything settled
CREATE VIEW revenue AS
SELECT order_date, region, revenue_gbp, 'batch' AS source
FROM   gold.revenue_batch
WHERE  order_date < CURRENT_DATE()
UNION ALL
SELECT order_date, region, revenue_gbp, 'speed' AS source
FROM   gold.revenue_realtime
WHERE  order_date = CURRENT_DATE();          -- batch has not settled this yet

-- the reconciliation nobody plans for, and everybody eventually writes:
SELECT b.order_date, b.region,
       b.revenue_gbp AS batch, s.revenue_gbp AS speed,
       ABS(b.revenue_gbp - s.revenue_gbp) AS drift
FROM   gold.revenue_batch b
JOIN   gold.revenue_realtime s USING (order_date, region)
WHERE  ABS(b.revenue_gbp - s.revenue_gbp) > 0.01;   -- alert on this

Five scenarios that between them produce all three architectures — including the hybrid nobody names, which is what most companies actually run.

Build them on the pipeline canvas with the chapter closed. Each is graded against the scenario’s own requirement rules rather than against a model answer, so there is more than one design that passes — and a design that does not pass is told exactly which requirement it missed.