APACHE AIRFLOWFoundations

Scheduling, Data Intervals & the Logical Date Trap

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

when does a run fire, and what period does it cover? Airflow 2 and Airflow 3 give different default answers to the second one, and almost all confusion lives in that gap.

3 min · Topic 1 of 8

This is the chapter that catches everyone, and it has become harder rather than easier — because Airflow 3 changed the default behaviour, so the classic explanation you will find in most tutorials is now describing a mode you may not be running.

By the end you should be able to look at a schedule, a start_date and a catchup setting and say exactly which runs will exist and when each one fires. Not approximately. Exactly.

Core mental model

A schedule answers two separate questions: when does a run fire, and what period does it cover? Airflow 2 and Airflow 3 give different default answers to the second one, and almost all confusion lives in that gap.

Why it matters

Scheduling mistakes do not raise errors. A DAG configured against the wrong mental model runs perfectly and processes the wrong day — which is the most expensive category of bug there is, because nothing alerts and the numbers look plausible. Chapter 1 called this silent success; this is where it comes from most often.

Logical date
The timestamp identifying a DAG run. Formerly execution_date, which was removed in Airflow 3. What it means depends on the timetable family.
Data interval
The window of time a run covers, from data_interval_start to data_interval_end. Under a trigger timetable the two are identical.
Timetable
The object that turns a schedule into runs. The two built-in families — trigger and data interval — behave differently.
Catchup
Whether Airflow creates runs for intervals between start_date and now, or only the most recent one.
Common mistake

Learning scheduling from a tutorial without checking which Airflow version it targets. Airflow 3 changed the default timetable for plain cron strings. Material written before 2025 describes behaviour you may not have, and the mismatch shows up as data landing under the wrong date rather than as an error.

Better habit

Always know which timetable family your DAG is using. It is one config setting and it changes what every date macro means.

Set schedule, start_date and catchup explicitly on every DAG. Defaults here are version- and config-dependent.

Before trusting a scheduling explanation, check whether it says data_interval_end is when the run fires. If it does, it is describing Airflow 2 semantics.

Two questions, not one

When does it fire, and what does it cover? Cron only ever answered the first. Airflow’s data intervals were an attempt to answer the second, and Airflow 3’s new default steps back toward cron.

Remember this

This is the highest-value chapter in the foundations. Slow down here and the rest of Airflow stops being surprising.