APACHE AIRFLOWAuthoring DAGs

Idempotency & Safe Task Design

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Idempotent means the number of times a task ran is not observable in its output. If you can tell, it is not.

⏱ 4 min · Topic 1 of 8

Chapter 12 gave you retries, clearing and backfills. Every one of them re-runs code that has already run at least once. This chapter is the assumption underneath all three: that running a task twice leaves the world in the same state as running it once.

Pick a write pattern below and run it more than once. Only one of the four gets the row count wrong, and it is the one most people write first.

Run the same task for the same date, more than onceSource has 1,000 rows for 2026-08-01. A correct task leaves 1,000 in the target, however many times it runs.
Times this task has run for 2026-08-011(a retry, a clear, or a backfill)
Expected rows1,000
Actual rows1,000
Not idempotent — correct for now, because it has only run oncePress + to run it again — a retry, a clear or a backfill — and watch the count. Every run adds the same 1,000 rows again. One retry doubles the day; a 90-day backfill after two failures triples it. Nothing errors, and the daily total is simply wrong.
INSERT INTO orders
SELECT * FROM staging_orders
WHERE ds = '{{ ds }}';
Core mental model

Idempotent means the number of times a task ran is not observable in its output. If you can tell, it is not.

Why it matters

This is not a theoretical property. Retries happen automatically, clearing happens during incidents, and backfills happen whenever history needs rebuilding. A task that is not idempotent corrupts data on every one of those events — silently, because each individual run succeeds.

Idempotent
Running it twice produces the same result as running it once. The precondition for every recovery mechanism in Airflow.
Partition
The slice of the target a run is responsible for — usually a date. Scoping writes to it is what makes replacement possible.
Atomic
A reader never sees a half-finished state. A different property from idempotency, solving a related problem.
Common mistake

Assuming a task is idempotent because it has never visibly broken. An append-only task looks perfect until the first retry. It then doubles a day of data, reports success, and the error is found weeks later by someone querying a total that does not reconcile.

Better habit

Ask of every write: what happens if this runs twice? If you cannot answer immediately, it is not safe.

Treat idempotency as a requirement of the task, not a property you hope for.

Verify by running twice and comparing, rather than by reasoning about the code.

The count must not be observable

If you can look at the output and tell how many times the task ran, it is not idempotent. That single test catches almost every case.

Remember this

Every recovery tool in Airflow re-runs your code. Idempotency is what makes that a recovery rather than a second failure.