Airflow Architecture: Scheduler, Executor, Workers, Metadata DB

Your shop runs one DAG every day. Its name is shop_orders_daily.

A DAG is a set of tasks with an order. This one has two tasks: load_orders and build_daily_revenue.

You have seen Airflow from the outside. Orchestration & Scheduling taught what an orchestrator does. Scheduling, Data Intervals & the Logical Date Trap taught when a run starts.

Monitoring, Logging, Alerting & Production Failures named some parts of Airflow. This chapter opens the machine and follows one run through every part.

You answer three questions:

1. Which part does which job?
2. Where is the state kept?
3. What happens when a part stops?

The run reads orders from these six. The run about 2026-01-20 reads order 105 and starts on 2026-01-21 00:00:

All times, ids and host names in this chapter are made up. Here is the DAG file. It is where the whole journey starts:

import pendulum
from datetime import timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.timetables.interval import CronDataIntervalTimetable

def load_orders():
    ...  # copy one day of orders

def build_daily_revenue():
    ...  # rebuild one day of daily_revenue

with DAG(
    dag_id="shop_orders_daily",
    schedule=CronDataIntervalTimetable("0 0 * * *", timezone="UTC"),
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
) as dag:
    load = PythonOperator(task_id="load_orders", python_callable=load_orders)
    build = PythonOperator(task_id="build_daily_revenue", python_callable=build_daily_revenue)
    load >> build

Airflow is not one program. It is several parts that work together. Each part has one job.

Look at the seven parts and their jobs:

Metadata means data about your pipelines. It is not your shop data. The orders table is never stored in this database.

In Airflow 2, the scheduler runs the DAG file processor by default. You can also run it as its own process. Airflow 3 always runs it on its own.

The executor is not a separate program. It runs inside the scheduler process. There are three main kinds:

• LocalExecutor starts each task as a process on the scheduler machine.
• CeleryExecutor sends each task through a broker to a group of workers. A broker is a message queue, such as Redis.
• KubernetesExecutor asks Kubernetes to start one pod for each task. Kubernetes is a system that starts and stops programs on many machines. A pod is a small, short-lived package of programs; here it runs one task.

The chapter Executors Compared: Local, Celery, Kubernetes compares them. Here you only need to know that an executor sends a task out to run.

Two parts are extra. You need the web server only if you want the UI or the API. You need the triggerer only for deferrable tasks.

Concurrency, Pools, Priorities & Deferrable Operators teaches deferrable tasks and the triggerer.

Only the metadata database keeps state. State means what is true right now: which runs exist and what each task is doing.

The other parts keep little in memory. When a part restarts, it reads the state back from the database.

Three tables matter most:

• serialized_dag holds the parsed form of each DAG, saved by the DAG file processor.
• dag_run has one row for each run of a DAG.
• task_instance has one row for each task in each run. Its state column is the main thing to watch.

You can read these tables with SQL. Look at the two task rows of our run at 00:00:12. The result is simplified:

SELECT task_id, state, hostname
FROM   task_instance
WHERE  dag_id = 'shop_orders_daily'
  AND  run_id = 'scheduled__2026-01-20T00:00:00+00:00';

Every part writes here or reads from here. The scheduler reads which tasks are ready, and a worker writes when its task ends.

Use Postgres or MySQL in production. SQLite is only for testing, because it cannot handle many parts writing at once.

The scheduler repeats one loop all day. Each pass has five steps:

1. Create runs that are due.
2. Find tasks whose upstream tasks, the tasks that must run first, are done.
3. Check for free room: a free pool slot, and room under the limits on how many tasks may run at once. A pool is a set of task slots.
4. Hand the tasks that fit to the executor.
5. Read what finished and update the states.

A task does not start the moment it is ready. It waits for the next pass, then for room, then for a worker. So Airflow suits minutes, not seconds.

A task instance is one task in one run.

Each task instance moves through states. The state tells you which part holds the task. Look at the five states:

Step 3 is where scheduled becomes queued. Concurrency, Pools, Priorities & Deferrable Operators explains pools.

This tells you where to look. A task stuck in scheduled points at pools and limits. A task stuck in queued points at workers.

Now follow the run about 2026-01-20 through every part. The times are made up. They are on 2026-01-21 UTC unless a cell shows a date.

Read each row as one step. The last column shows the new state:

Look at what is missing. No part calls the next part directly. Each step ends with a write to the database, and the next step starts by reading it.

That is why a part can restart in the middle of a run. The state is in the database, not in the part.

The last step is to read the final rows. This is the same query as before, with more columns:

SELECT task_id, state, queued_dttm, start_date, end_date, hostname
FROM   task_instance
WHERE  dag_id = 'shop_orders_daily'
  AND  run_id = 'scheduled__2026-01-20T00:00:00+00:00';

The tasks did very little work. Order 105 is the only order of 2026-01-20, and it is pending, so that day has no revenue.

People mix up these two. The executor is how the scheduler sends a task out. The worker is the place where the task runs.

In the timeline, the executor acted at 00:00:03. Worker worker-a acted at 00:00:04. The executor is a setting in the scheduler, not a service you deploy.

You choose it with one line in the config. Look at the line:

# airflow.cfg
[core]
executor = CeleryExecutor

So what do people mean when they say the executor has a problem? Look at the table:

With LocalExecutor, tasks run as children of the scheduler. If the scheduler stops, those tasks stop too. With Celery or Kubernetes, running tasks live on other machines.

Executors Compared: Local, Celery, Kubernetes shows how to choose between them.

Each part fails in its own way. Some failures are loud. Others are quiet: nothing fails, but nothing moves.

This table assumes Airflow 2 with Celery or Kubernetes workers. Look at what each stop does:

A worker can die in the middle of a task. The row still says running, but no heartbeat comes. After a timeout, the scheduler sees this.

This is a zombie task. The scheduler marks it failed. Because our DAG has retries, it goes to up_for_retry and runs again.

Look at the rows of build_daily_revenue if worker-b died at 00:00:12. The states are what you would see over time:

One scheduler is a single point of failure. Since Airflow 2.0, you can run more than one scheduler at the same time.

They have no leader. Each one locks the task rows it works on, so two schedulers never take the same task. This needs Postgres or MySQL 8, not SQLite.

More schedulers do not fix every limit. Three limits matter most as you grow:

• Parse time. Every DAG file is read again and again. Slow code at the top of a file slows everything.
• Database connections. Every scheduler, worker and UI request opens connections. A connection pooler such as PgBouncer shares a few connections between many users.
• Database size. Old runs, task rows and event rows make the tables big. The command airflow db clean removes old rows.

These limits show up as one symptom: everything gets slower at once. No single DAG looks broken.

Watch the dag_processing.total_parse_time metric and the database connection count. Monitoring, Logging, Alerting & Production Failures covers metrics.

Airflow 3 moves some jobs between parts. The goal is more isolation. Code that you wrote should not run next to the scheduler.

Three changes matter here:

• The API server replaces the web server. It also serves the API that running tasks use.
• The DAG file processor always runs as its own process. The scheduler no longer reads your DAG files.
• A running task reports its state to the API server. It no longer connects to the metadata database.

The shape of our run stays the same. The states and the five loop steps are still there.

One result is new. If the API server stops, running tasks cannot report their state. In Airflow 2, a stopped web server cost you only the UI and the REST API.

The chapter Managed Airflow, Airflow 3 & Migration covers how to move from Airflow 2 to Airflow 3.

Airflow is several parts around one database. The parts do not call each other. They write and read rows.

Look at the answers you should know:

Try this alone: name the part that acts and the new state of load_orders at 00:00:02, at 00:00:04 and at 00:00:07.

The answers are the scheduler and queued, a worker and running, then a worker and success.