DAGs, Tasks & Operators: The Core Model

Airflow uses four words that sound alike. Two name things you write, and two name things Airflow creates:

So the shop has one DAG with 3 tasks. After 7 daily runs, it has 7 DAG runs and 3 x 7 = 21 task instances.

A task cannot fail; it is only a description. A task instance can fail. Say "the task instance for 2026-01-20 failed", and people know exactly where to look.

DAG means directed acyclic graph. Directed means each arrow points one way: load_orders >> build_daily_revenue means build waits for load. Acyclic means no path leads back to where it started.

Here is a DAG that breaks the rule. The last line makes load_orders wait for build_daily_revenue, which waits for load_orders:

load_orders = EmptyOperator(task_id="load_orders")
build_daily_revenue = EmptyOperator(task_id="build_daily_revenue")

load_orders >> build_daily_revenue >> load_orders

Python builds the objects without a complaint. Airflow then checks the DAG and refuses it, so the DAG never runs. The UI shows it as an import error.

Why the rule? To start, the scheduler must find a task whose upstream tasks are all done. In a loop, every task waits for another one, so no task can ever start.

Most loops people want are something else:

You can write a task in three ways:

Underneath, all three are operators. A @task function becomes an operator class too. In the graph, each is one task, with one task instance per run.

A provider package holds the operators for one outside system, such as Snowflake or AWS. Check for one before you write your own.

Each task instance has a state. The state tells you which part of Airflow to check. These are the states you meet most:

Two pairs are often mixed up. scheduled and queued are both before running. But a task stuck in queued means its upstream is fine, and the problem is the workers or the executor.

failed and upstream_failed also differ. When a run has one failed task and ten upstream_failed ones, read the log of the failed one. The others never ran.

Airflow does not read your DAG file once. A part called the DAG processor runs the file again every few minutes or seconds, to notice your changes. To parse a file is to read and run it like this.

So every line at the top level of the file runs on every parse. Top level means outside any task. Compare these two versions:

# Wrong: runs on every parse, all day
import pandas as pd
TABLES = Variable.get("tables", deserialize_json=True)

# Right: runs once, when the task itself runs
@task
def load_orders():
    import pandas as pd
    tables = Variable.get("tables", deserialize_json=True)
    ...

Both versions give the same result when the task runs. But the first one slows down parsing for the whole Airflow, not only for your DAG.

Keep the top level for imports Airflow needs, fixed values and the DAG itself. Put all real work inside tasks.

Two tasks of one DAG can run on different machines, minutes apart. They share no memory and no local files.

A DAG file looks like one Python program, so this mistake is easy:

ROWS = []                        # each worker has its own copy

@task
def load_orders():
    ROWS.extend(fetch_orders())  # fills this worker's copy

@task
def build_daily_revenue():
    write_revenue(ROWS)          # another worker: ROWS is empty

Pass a small reference instead, such as a file path or a table name. The data stays in storage, which every worker can reach. Chapter 8 shows how, with XComs.

Three rules prevent most beginner problems:

1. The graph has no loops. Use retries, mapping, schedules or sensors instead.
2. The top level of a DAG file holds definitions only. Work goes inside tasks.
3. Tasks share nothing. Pass references, and keep data in storage.

Check yourself: a run has one failed task and ten upstream_failed tasks. Whose log do you read? (Answer: the failed one, because the others never ran.)