Backfills, Retries & Deadline Alerts

Things go wrong in every pipeline. This chapter answers four questions, each with its own setting:

Most repairs here run tasks again. So they only work if a task is safe to run twice, which is the topic of chapter 13.

Many failures fix themselves: a short network problem, a busy API, a database that is still starting. Retries try the task again after a wait:

@dag(
    ...,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
)
def shop_orders_daily():

    @task                          # gets 2 retries from default_args
    def load_orders(): ...

    @task(retries=0)               # never retry: it would email twice
    def email_report(): ...

Retries delay the alert. on_failure_callback runs only after the last try fails. We tested a task with 2 retries:

Choose the number by the cause. A flaky network gets 3 to 5 retries. A bug in your code fails the same way every time, so retries only delay the alert.

For a busy API, add retry_exponential_backoff=True. Each wait is then longer than the one before, up to max_retry_delay.

Airflow has several time limits, and each one covers something different:

execution_timeout limits one try, not the task. Take execution_timeout of 30 minutes, 3 retries and a retry_delay of 10 minutes.

The worst case is 4 tries of 30 minutes, plus 3 waits of 10 minutes. That is 150 minutes before the task finally fails.

Only dagrun_timeout limits the whole run. Set it on every production DAG, so no run stays open forever.

Callbacks run when a task reaches a state. Choose the one that fits the problem:

Callbacks do not help when a run is slow but not failing. Airflow 2 had SLAs for that. Airflow 3.0 removed them, and Airflow 3.1 added deadline alerts instead.

from airflow.sdk.definitions.deadline import AsyncCallback, DeadlineAlert, DeadlineReference
from shop.alerts import tell_team      # an async function

@dag(
    ...,
    dagrun_timeout=timedelta(hours=4),
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(hours=2),
        callback=AsyncCallback(tell_team),
    ),
)

The alert comes after 2 hours, while the run is still going. The hard stop comes after 4 hours, from dagrun_timeout.

A backfill creates runs for past dates. Use it when those runs never existed, for example for a new DAG, or after catchup=False skipped some days (chapter 5).

In Airflow 3, the CLI only creates the backfill. The scheduler then runs it, with the same pools and limits as normal runs:

airflow backfill create --dag-id shop_orders_daily \
  --from-date 2026-01-10 --to-date 2026-01-12 \
  --max-active-runs 1

--reprocess-behavior decides what happens to dates that already have a run. In our test, 2026-01-12 already had a successful run:

Each problem has its own repair. Pick it from what you see:

Clear repairs runs that exist. Backfill creates runs that do not exist. Mark success only records a fact, so use it only when the fact is true.

Marking a failed task as success hides the failure. The tasks after it run on data that was never made.

Try, stop, tell, repair:

1. Retries for causes that pass. No retries for bugs or for tasks that email.
2. execution_timeout limits one try. dagrun_timeout limits the whole run.
3. on_failure_callback fires after the last try. Deadline alerts replace SLAs.
4. Backfill with --max-active-runs, and choose --reprocess-behavior on purpose.
5. Clear for runs that exist, backfill for runs that do not.

Check yourself: execution_timeout is 20 minutes, retries is 2, and retry_delay is 5 minutes. What is the worst case? (Answer: 3 x 20 + 2 x 5 = 70 minutes.)