APACHE AIRFLOWFoundations

DAGs, Tasks & Operators: The Core Model

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Easy
01 · Orientation

What You’ll Master Here

DAG and task are nouns you write. DAG run and task instance are nouns Airflow creates. You edit the first pair; you operate on the second pair.

⏱ 4 min · Topic 1 of 8

Chapter 2 named the components. This chapter names the objects those components move around — and the reason it gets its own chapter is that four of them have similar names and are constantly used interchangeably by people who should know better.

Get this vocabulary precise and the rest of Airflow becomes readable. Leave it fuzzy and you will misdescribe your own bugs, which is the fastest way to get bad help.

Core mental model

DAG and task are nouns you write. DAG run and task instance are nouns Airflow creates. You edit the first pair; you operate on the second pair.

Why it matters

Almost every confusing Airflow conversation is a vocabulary problem in disguise. “The task failed” is ambiguous — a task is a definition and cannot fail; a task instance can. “I fixed the DAG” does not explain whether the fix applies to runs that already exist. Precision here saves entire debugging sessions.

DAG
The workflow definition — the graph of tasks you wrote. There is exactly one, no matter how long it has been running.
DAG run
One execution of that DAG for one data interval, with its own state and its own set of task instances.
Task
A node in the graph — also a definition. It describes work; it does not do work.
Task instance
One task, in one DAG run, on one attempt. The thing that actually has a state, a log and a duration.
One DAG file. How many objects does it actually create?
Tasks in the DAG3
Days it has run for4
1DAGThe definition. One, no matter how long it runs.
4DAG runsOne per data interval that has closed.
3TasksAlso definitions — nodes in the graph.
12Task instancesOne task, in one run, on one attempt.
3 tasks × 4 runs = 12 task instancesOnly the last number grows. When people say “Airflow is slow” or “the database is huge”, this is almost always the number they mean — and retries multiply it further, because each attempt is another row.
Editing the DAG file changes the two definitions on the left. It does not retroactively change the runs that already happened — which is exactly why Airflow 3 added DAG versioning.
Common mistake

Using “task” and “task instance” interchangeably. You end up saying things like “the task is stuck in queued”, which is not possible — definitions do not have states. The imprecision hides whether the problem is in the code you wrote or in one particular run.

Better habit

Say “task instance” when you mean something with a state. It is a small habit that makes every incident description sharper.

When someone reports a failure, establish which run and which attempt before looking at anything.

Remember that editing the DAG file changes definitions only. Existing runs keep the shape they were created with.

Definitions versus instantiations

This is the whole chapter in one line. A DAG is instantiated into a DAG run each time it runs, and the tasks under it are instantiated into task instances. Same relationship, one level down.

Remember this

Two things you write, two things Airflow creates. Keeping them straight is not pedantry — it is how you describe a bug accurately.