Lambda, Kappa & Streaming-First

You want numbers that are fresh and also correct. This chapter compares the two famous designs that try to give you both.

What you already know

• Batch vs Streaming vs Micro-Batch: batch runs on stored data now and then. Streaming handles each event as it arrives.
• Streaming Architecture Fundamentals: Kafka keeps events in a log. The log has a retention time, and a reader can replay it from an offset.
• Stream Processing and Delivery Guarantees & Ordering: a stream job can crash and repeat work. You make the repeat harmless.
• Incremental Processing & Backfills: you can fix old results by running the job again over old data.

What you will learn

1. Lambda: a batch path and a speed path, with one merged answer.
2. Kappa: one streaming path. You fix old results by replaying the log.
3. Streaming-first: one engine and one table for batch and stream.
4. How to choose, and what each design costs.

We use the same shop and the same made-up day, 2026-01-22. We add one goal: total sales by country. Here are the orders and the customers:

-- Goal: total sales by country
-- Orders are joined to customers on customer_id.
-- A customer with no country is shown as 'unknown'.

Here the topic orders has one partition, to keep the example easy. So offsets 0 to 6 follow arrival order, as in the one-log example of Streaming Architecture Fundamentals.

Nathan Marz named the Lambda design. It runs two paths over the same data and merges their answers. Each path is called a layer.

• The batch layer reads all stored orders and rebuilds the numbers every so often. It is correct but slow.
• The speed layer is a stream job. It handles only the orders that came after the last batch run. It is fast.
• The serving layer is what readers query. It adds the batch view and the speed view together.

Say the batch layer last ran at 10:06 and read the orders that had arrived by then. The speed layer handled every order after that. Now it is 10:10:

-- Batch layer (runs at 10:06): only orders that arrived by 10:06
SELECT COALESCE(c.country, 'unknown') AS country, SUM(o.amount) AS sales
FROM orders o JOIN customers c ON c.customer_id = o.customer_id
WHERE o.arrives <= '10:06'
GROUP BY 1;

-- Speed layer: a stream job over orders that arrive after 10:06.
-- It adds each new amount to a running total per country.

-- Serving layer, at 10:10: batch view + speed view, per country.

The batch layer saw orders 110, 111 and 112. The speed layer saw 113, 114, 150 and 151. Order 113 happened at 10:04 but arrived at 10:07, so the batch run missed it and the speed layer caught it.

At 10:12, the batch layer runs again over all seven orders. Its new result replaces the speed result for those orders. From then on, the batch answer is the one to trust, and the speed layer counts only newer orders.

The cost: two code paths

The batch code and the speed code are two programs. They must do the same thing. Say the speed code is written by another person, and it skips a customer with no country:

# Speed layer code (stream job)
for order in new_orders:
    country = country_of(order.customer_id)
    if country is None:
        continue                  # skips the order: a bug
    running_total[country] += order.amount

# Batch layer code (SQL): COALESCE(country, 'unknown') keeps the order.

Readers saw 97.00, and then the number jumped to 104.50. The batch layer fixed it, but only by surprise. This is the main cost of Lambda: two programs, two sets of tests, and answers that must match.

Jay Kreps described Kappa in 2014, in an article called "Questioning the Lambda Architecture". It keeps only the streaming path. There is no second program.

To fix old results, you do not run a batch job. You start a new version of the stream job and replay the log from the start. It writes to a new output table.

Say the first version of the job skips customers with no country. We found the bug. This is the fix:

# v1 runs now and writes sales_by_country_v1 (it has the bug).

# v2 has the fix. It reads the whole log and writes a NEW table.
start_stream(
    source="topic orders",
    start_at_offset=0,              # replay from the first event
    logic=sales_by_country_v2,      # missing country -> 'unknown'
    output="sales_by_country_v2")

# Step 1: v1 keeps running while v2 catches up.
# Step 2: when v2 reaches the end of the log (lag 0), compare v1 and v2.
# Step 3: point readers to v2. Step 4: stop v1 and drop its table.

You changed one program and replayed. Readers kept using v1 until v2 was ready. Compare this with Lambda, where you fix two programs.

What Kappa needs

• A log that still holds the old events. Use a long retention time, or tiered storage, which moves old log data to cheaper storage.
• An engine that can process old events fast. A replay is a big job, not a trickle.
• An output you can write to a new table, and readers you can switch.

A compacted topic does not meet the first need. It keeps only the newest event for each key, so it cannot rebuild a sum from all events. Now see what a short retention does:

-- Retention has deleted offsets 0 and 1 (orders 110 and 111).
-- The log now starts at offset 2.
-- A replay set to start at the earliest event really starts at offset 2.

Many teams today run a third shape. It is called streaming-first. It is Kappa, but the same engine and the same code also run as a batch job.

• Lambda: two programs, and the batch answer replaces the speed answer.
• Kappa: one stream program. You reprocess by replaying the log.
• Streaming-first: one program for batch and stream, writing to a table that both read.

Spark Structured Streaming and Flink can run the same logic over stored data or over a stream. In Spark, you write the logic once, as a function. Then you give it either a batch input or a stream input:

from pyspark.sql import functions as F

def sales_by_country(orders, customers):
    joined = orders.join(customers, "customer_id", "left")
    with_country = joined.withColumn(
        "country", F.coalesce("country", F.lit("unknown")))
    return with_country.groupBy("country").agg(F.sum("amount").alias("sales"))

# Batch: read all stored orders, then stop.
batch_result = sales_by_country(spark.read.table("orders"), customers)

# Stream: the same function, on orders as they arrive.
stream_result = sales_by_country(spark.readStream.table("orders"), customers)

In Spark, a trigger sets when the stream reads. With the trigger availableNow (Spark 3.3 or later), the stream reads all waiting data, then stops. So the same stream job can also run like a batch job, as Structured Streaming Fundamentals shows.

A table as the serving layer

In this shape, the output is a table in a format like Delta Lake. The stream writes one row per order to it, and dashboards add up the rows. A batch job can fix rows with MERGE, which updates rows that match and inserts rows that do not.

The old stream code skipped order 114, because customer 4 has no country. A batch job runs the fixed code over the stored orders and makes the missing row:

MERGE INTO order_sales AS t
USING fixed_rows AS f
  ON t.order_id = f.order_id
WHEN MATCHED THEN UPDATE SET t.country = f.country, t.amount = f.amount
WHEN NOT MATCHED THEN INSERT (order_id, country, amount) VALUES (f.order_id, f.country, f.amount);

Do not start from a name. Start from four questions about your own data.

1. How fresh must the numbers be: seconds, minutes, or hours?
2. How often will the logic change? Each change means a fix, and a fix means a reprocess.
3. How long can you keep the log? Can a replay reach the first event?
4. Does your team already run a streaming engine?

Then compare the options. This table shows when each one fits and what it costs:

Use the questions on three cases from the shop. They have different answers:

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.