What You’ll Master Here
who wrote that row, and who is supposed to read it next?
Chapter 1 argued that an orchestrator runs work based on readiness rather than the clock. This chapter opens the machine that does it. By the end you should be able to draw Airflow on a whiteboard from memory and say what each box would do if you unplugged it.
That second half is the part that matters on call. Almost every Airflow incident is a component behaving exactly as designed while somebody misreads which component is at fault — a fix that “does nothing” is usually the dag processor, and a UI that looks fine while nothing runs is usually the scheduler.
Airflow is not one process. It is a handful of stateless services around one stateful database, and every question about behaviour reduces to: who wrote that row, and who is supposed to read it next?
You cannot debug a distributed system you think is a single program. Engineers who picture Airflow as “the thing that runs my DAGs” get stuck the first time a task sits in queued forever, because that symptom points at four different components and only the architecture tells you which one to check.
- Scheduler
- The component that triggers scheduled workflows and submits ready tasks to the executor. It also runs the executor inside its own process.
- Dag processor
- The component that parses your DAG files from a dag bundle and serialises them into the metadata database. In Airflow 3 it is always a standalone process.
- Metadata database
- Postgres or MySQL, holding the state of every DAG, run, task instance, connection and variable. The single source of truth.
- API server
- Serves the REST API and the UI — and in Airflow 3 also receives state from running tasks, so workers never touch the database.
- Owns: Decides what is ready to run and hands it to the executor. Creates DAG runs on schedule.
- Worth knowing: The executor lives inside the scheduler process — it is a configuration property, not a separate service you start.
Treating Airflow as one process that either “is running” or “is not”. You restart everything for a problem that lived in one component, lose the running tasks along with it, and learn nothing about the cause — so it happens again next week.
When something is wrong, name the component before naming the fix. “Nothing is scheduling” and “my change did not appear” are different components entirely.
Remember which components are stateless. Anything except the database can be restarted; the state survives because it was never in memory.
Learn the architecture of the Airflow version you actually run. The 2 → 3 changes move real responsibilities between components.
This is the sentence to hold on to. The metadata database is the system; everything else is a worker process that reads intent out of it and writes results back.
Airflow is a small distributed system pretending to be a tool. Learn the boxes now and every later chapter gets easier.