What You'll Master Here
Think in partitions. Process new partitions incrementally; reprocess old partitions to backfill or fix; replace a partition when late data lands. Idempotency makes every reprocess safe.
August is short. A currency bug shipped eleven days ago, so eleven days of numbers are wrong and the rest are fine. Fixing them means running a pipeline you wrote for "today" over dates in the past — without touching the days that were already correct.
That is this chapter: incremental processing (do only the new work each run) and backfilling (redo old work on purpose). Both rest entirely on the idempotency from Chapter 11 — if a re-run is not safe, none of this is available to you.
The unifying concept is the partition: a named slice of data, almost always a time window like a single day. Partitions are the unit you process incrementally, the unit you reprocess in a backfill, and the unit you replace when late data arrives. Once you think in partitions, incremental runs and backfills stop being scary and become routine.
By the end you will be able to design a pipeline that processes only what changed, backfill a range of history safely without double-counting, and handle late-arriving data without corrupting past results. These are the operations that separate a pipeline you can confidently change from one nobody dares touch.
Think in partitions. Process new partitions incrementally; reprocess old partitions to backfill or fix; replace a partition when late data lands. Idempotency makes every reprocess safe.
Pipelines are not write-once: requirements change, bugs are found, data arrives late. The ability to reprocess history safely is what lets a pipeline evolve. Without it, every fix is a terrifying manual operation; with it, correcting the past is a single, safe command.
- partition
- A named slice of a dataset, usually a time window (e.g. one day), processed as a unit.
- incremental processing
- Processing only the new or changed data each run, not the whole history.
- backfill
- Re-running the pipeline over past partitions to populate or correct history.
- late-arriving data
- Events that show up after the partition they belong to was already processed.
Designing a pipeline that can only ever process "now" going forward. When a bug is found or logic changes, there is no safe way to fix history; the past stays wrong.
Partition data (usually by time) as the unit of processing.
Make every partition reprocessable and idempotent.
Plan for backfills and late data from the first design.
Partitions plus idempotency turn "reprocessing the past" from a dangerous one-off into a routine operation. Design for reprocessing and you can change a pipeline without fear.
Learn partitions first, then incremental processing, then backfills and late data. Notice that all three rest on the same foundation: an idempotent, partition-scoped run.
Incremental processing and backfills both operate on partitions and rely on idempotency; thinking in reprocessable partitions is what lets a pipeline safely evolve and correct its history.
- Define a partition and give a typical example.
- Explain why a pipeline must be able to reprocess the past, not just go forward.