Incremental Processing & Backfills

August is short. A currency bug shipped eleven days ago, so eleven days of numbers are wrong and the rest are fine. Fixing them means running a pipeline you wrote for "today" over dates in the past — without touching the days that were already correct.

That is this chapter: incremental processing (do only the new work each run) and backfilling (redo old work on purpose). Both rest entirely on the idempotency from Chapter 11 — if a re-run is not safe, none of this is available to you.

The unifying concept is the partition: a named slice of data, almost always a time window like a single day. Partitions are the unit you process incrementally, the unit you reprocess in a backfill, and the unit you replace when late data arrives. Once you think in partitions, incremental runs and backfills stop being scary and become routine.

By the end you will be able to design a pipeline that processes only what changed, backfill a range of history safely without double-counting, and handle late-arriving data without corrupting past results. These are the operations that separate a pipeline you can confidently change from one nobody dares touch.

partition

A named slice of a dataset, usually a time window (e.g. one day), processed as a unit.

incremental processing

Processing only the new or changed data each run, not the whole history.

backfill

Re-running the pipeline over past partitions to populate or correct history.

late-arriving data

Events that show up after the partition they belong to was already processed.

A partition is a named, independent slice of a dataset, and in pipelines it is almost always a time window: one day, one hour. Partitioning by date means each day’s data lives in its own logical bucket that can be read, written, and replaced on its own, without touching any other day.

This independence is the key property. Because 2026-06-03 is a self-contained partition, you can reprocess just that day, fix just that day, or overwrite just that day, while every other day stays untouched. Combine partition-scoped processing with the partition-overwrite idempotency from Chapter 9, and reprocessing any single partition becomes a safe, identical replacement.

Partitioning also drives performance: a query for "yesterday" reads only yesterday’s partition, not the whole table (partition pruning). But for this chapter the crucial role is operational, the partition is the unit you process incrementally, the unit you backfill, and the unit you replace for late data. Everything else in the chapter builds on it.

partition key

The column that defines partitions, usually a date (event_date) or hour.

partition independence

The property that one partition can be processed or replaced without affecting others.

partition pruning

Skipping irrelevant partitions in a query, reading only the ones needed.

partition overwrite

Replacing a single partition wholesale, the idempotent way to reprocess it.

Incremental processing means each run handles only the new or changed data, not the entire history. A daily pipeline processes today’s partition; it does not recompute all of last year every night. This is the difference between a pipeline that stays fast as data grows and one that gets slower until it no longer fits its window.

There are two flavours, and they map to ideas you have already met. Append-style incremental processes a new partition each run (today’s events) and writes it, ideal for immutable, time-stamped data. Merge-style incremental processes changed records since a watermark (Chapter 7) and upserts them, ideal for mutable records like orders that update.

The state is the whole risk. Full refresh is simple and self-healing and does not scale; incremental is efficient and has to remember where it got to. The lab below crosses the two ways of writing that memory down with the two ways of writing the rows — one of the four cells loses data permanently and nothing anywhere reports it.

append-style incremental

Processing and adding a new partition each run; for immutable, time-stamped data.

merge-style incremental

Processing changed records since a watermark and upserting them; for mutable records.

processing state

The record of what has already been processed (a watermark or processed-partition list).

full refresh

Reprocessing the entire dataset; simple and self-healing but does not scale.

SELECT * FROM orders WHERE updated_at >  '2026-03-14 09:00:00.000'

A backfill re-runs the pipeline over past partitions. You backfill to populate history when a new pipeline goes live (compute the last two years), or to correct history after a bug or logic change (recompute March now that the rule is fixed). It is one of the most common real operations, and one of the most dangerous if the pipeline was not designed for it.

The diagram shows the shape: while normal daily runs continue on the latest partition, a backfill reprocesses a range of earlier partitions (here 06-01 to 06-03). Because each partition is independent and the load is idempotent (overwrite the partition), reprocessing a past day replaces it cleanly, no duplicates, no effect on other days. This is exactly why Chapters 9 and 11 spent so long on idempotency.

The cardinal rule is therefore: never backfill a non-idempotent pipeline. A backfill is just many re-runs, so if a single re-run duplicates data, a backfill multiplies the damage across every partition in the range. Backfill safety is not a property of the backfill command; it is a property of the pipeline being idempotent and partitioned.

backfill

Re-running a pipeline over historical partitions to populate or correct them.

partition-scoped reprocess

Reprocessing one partition at a time so each is replaced cleanly and independently.

backfill range

The set of past partitions a backfill covers (e.g. all of March).

historical correctness

Ensuring past partitions reflect the current, correct logic after a fix.

Data is not always punctual. An event that happened at 11:58 PM on Monday might not reach your pipeline until Tuesday, because of a mobile device that was offline, a delayed upstream system, or a network retry. This is late-arriving data, and it is one of the subtlest correctness problems in pipelines.

The danger is that a naive incremental pipeline already "closed" Monday’s partition and moved on, so Monday’s late event either lands in Tuesday (wrong day) or is dropped entirely. Either way, Monday’s numbers are quietly wrong. The fix builds directly on partitions and idempotency: route the late event to the partition it belongs to (Monday) and reprocess that partition, overwriting it with the corrected total.

Because reprocessing a partition is idempotent, handling late data is a targeted mini-backfill of the day it belongs to. What is left is one number: how long you leave the door open. Move it below and watch three things move with it — completeness, the number of partitions that can still change, and whether finance can ever close a month.

event time vs processing time

When an event actually happened versus when the pipeline received it; late data is when they diverge.

allowed lateness

How long a pipeline keeps accepting and reprocessing late data for a partition.

partition reprocess

Overwriting the affected partition to incorporate late events, a targeted backfill.

finalisation

Declaring a partition final after the lateness window, so it stops being reprocessed.

On the 3rd, a currency conversion started rounding the wrong way. Nobody noticed until the 14th, when finance asked why August was short.

Before writing a single command you have to answer three questions: which partitions are wrong, in what order they rebuild, and how fast you dare go. The grid below answers the first; the code answers the rest.

blast radius

The set of partitions a bug actually touched. Derived from the data rather than the deploy date, because a bug often needs a particular input to trigger.

lateness window

How many trailing partitions each run rebuilds. It is what lets a Monday event arriving on Tuesday reach Monday’s numbers.

restatement

Changing what a metric means and rebuilding history to match. A communication exercise with a backfill attached, not the other way round.

completion marker

A per-date record that the rebuild finished, so an interrupted backfill resumes rather than starting again.

A currency-conversion bug · found on the 14thBefore you can fix history you have to know how much of it is wrong.12 partitionsReset

Which day did the bad code ship?

the 3ththe 8ththe 12th

How many partitions may rebuild at once?

1412

August · silver.orders, one box per date partition

0102030405060708091011121314

Red partitions were built by the broken code. Everything before the 3th is correct and must not be touched — rebuilding a good partition is how a backfill creates new bugs.

What has to be rebuilt, and in what order

marlow-bronzeRAWUntouched. Never rebuilt.→marlow-silverREBUILDWhere the bug lives.→marlow-goldREBUILDDerived from silver.→finance reportRE-READRestated once gold lands.

12 silver partitions12 gold partitions3 waves~18 min

12 partitions, 4 at a time, about 18 minutesBronze is never rebuilt: it is what the rebuild reads from. Silver is rebuilt per date, then gold, then the report is restated. Consumers keep reading the old numbers until each partition swaps.

-- when did the numbers start disagreeing with the source of truth?
SELECT dt,
       SUM(total_gbp)                              AS warehouse_total,
       SUM(raw_total * fx_rate)                    AS recomputed_total,
       ABS(SUM(total_gbp) - SUM(raw_total * fx_rate)) AS drift
FROM   silver.orders  s
JOIN   bronze.orders  b USING (order_id, dt)
JOIN   silver.fx_rates f ON f.ccy = b.currency AND f.dt = b.dt
WHERE  dt >= DATE '2026-07-25'          -- look back well before the suspected date
GROUP  BY dt
HAVING drift > 0.01
ORDER  BY dt;                            -- the first row is your true start date

# Airflow: 11 dates, four at a time, into a pool that caps source connections
airflow dags backfill marlow_orders_daily   --start-date 2026-08-03 --end-date 2026-08-13   --max-active-runs 4   --reset-dagruns

# dbt: the same range, and --full-refresh because the incremental
# predicate would otherwise skip dates that already have rows
dbt run --select silver.orders+ --full-refresh   --vars '{"start_date": "2026-08-03", "end_date": "2026-08-13"}'

# Plain Python, when the orchestrator is not involved: one date at a time,
# and a marker so an interrupted backfill resumes instead of restarting
for dt in date_range("2026-08-03", "2026-08-13"):
    if backfill_log.done(dt):
        continue
    rebuild_partition(dt)          # overwrite, never append
    backfill_log.mark_done(dt)

-- rebuild the last 3 days every run, not just yesterday:
-- a Monday event arriving on Tuesday still lands in Monday's partition
{{ config(materialized='incremental', unique_key=['dt','order_id'],
          partition_by={'field': 'dt', 'data_type': 'date'},
          incremental_strategy='insert_overwrite') }}

SELECT order_id, customer_id, total_gbp, dt
FROM   {{ ref('bronze_orders') }}

{% if is_incremental() %}
  WHERE dt >= DATE_SUB(CURRENT_DATE(), INTERVAL 3 DAY)
{% endif %}

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

for dt in dates_to_rebuild:                 # sequential, or a bounded pool
    day = (spark.read.parquet("s3://marlow-bronze/orders/")
                .where(col("dt") == dt))    # read raw, not the broken silver

    silver = transform(day)                 # the FIXED logic

    (silver.write
        .mode("overwrite")                  # + dynamic mode = this date only
        .partitionBy("dt")
        .parquet("s3://marlow-silver/orders/"))

Five scenarios about the partition as the unit of reprocessing — forward as an incremental load, backward as a backfill that must not break tonight.

Build them on the pipeline canvas with the chapter closed. Each is graded against the scenario’s own requirement rules rather than against a model answer, so there is more than one design that passes — and a design that does not pass is told exactly which requirement it missed.