What You'll Master Here
You want fresh and accurate at once. Lambda gets there with two paths (batch + stream) merged; Kappa with one streaming path that you replay to reprocess.
This chapter zooms out from individual streaming mechanics to whole-system architecture. The central question is timeless: you want data that is both fresh (streaming) and complete/accurate (batch), so how do you combine them? The two famous answers are the Lambda and Kappa architectures, and the modern streaming-first view.
Lambda runs batch and streaming side by side and merges their results. Kappa argues that is needless duplication and keeps a single streaming path, reprocessing by replaying the log. The debate between them is really a debate about complexity versus capability, and it teaches you how to think about freshness and reprocessing at the architecture level.
By the end you will be able to draw both architectures, explain the "two codebases" problem that motivated Kappa, reason about how reprocessing works in each, and make a sober recommendation, which, as usual, favours the simplest design that meets the requirement.
You want fresh and accurate at once. Lambda gets there with two paths (batch + stream) merged; Kappa with one streaming path that you replay to reprocess.
These architectures frame how real companies reconcile real-time and historical correctness. Understanding the trade-off lets you avoid the classic mistake of building and maintaining two parallel systems when one would do, or forcing everything into streaming when batch is simpler.
- Lambda architecture
- A design with a batch layer (accurate) and a speed layer (fast), merged into one serving view.
- Kappa architecture
- A streaming-only design that reprocesses history by replaying the log through the same code.
- reprocessing
- Recomputing results after a logic change or bug, the operation that distinguishes the two.
- serving layer
- Where query-ready results live for consumers, fed by the architecture above it.
Adopting Lambda’s two-path complexity without needing both layers. You maintain and reconcile two codebases forever for freshness or accuracy you could get more simply.
Frame the choice as freshness + accuracy versus complexity.
Prefer one path (Kappa-style) when a single engine can meet the need.
Treat reprocessing as a first-class design requirement.
The whole debate is "one path or two?". Lambda accepts two paths for capability; Kappa keeps one for simplicity. The right answer is the simplest architecture that delivers the required freshness and accuracy.
Read Lambda, then Kappa, then reprocessing (the crux), then choosing. The reprocessing topic is what really separates the two architectures.
Lambda and Kappa are two answers to "fresh and accurate at once": two merged paths versus one replayable stream; the choice is capability versus complexity.
- State the core difference between Lambda and Kappa in one sentence.
- Explain why wanting both freshness and accuracy creates this architectural problem.