Why Orchestration Matters & Where Airflow Fits

Every night, the shop must load the new orders, build the daily revenue table, and refresh the dashboard. Each step needs the one before it.

An orchestrator is a tool that runs steps like these in the right order. It retries them when they fail, and records what happened. Airflow is the most common orchestrator for data work.

This chapter shows why a simple timer is not enough, what an orchestrator gives you, and when Airflow is the wrong tool. The Data Pipeline chapter Orchestration & Scheduling covers the same idea without Airflow.

cron is a program on Linux that starts a command at a set time. A crontab is the file that lists those commands and times. Here is the shop job, written as three cron lines:

# minute hour day month weekday  command
  0      2    *   *     *        python load_orders.py
 30      2    *   *     *        python build_daily_revenue.py
  0      3    *   *     *        python refresh_dashboard.py

Nothing in this file says that build needs load. The 30 minutes is a guess: load usually takes 15 minutes.

Now one bad night. The orders source is slow, and load_orders takes 70 minutes.

At 02:30, build_daily_revenue starts anyway. It reads yesterday's orders, builds the table, and ends with no error.

At 03:00 the dashboard shows the old revenue under today's date. Every job reported success, so nobody gets an alert. Someone finds the wrong number in the morning.

cron is good at one thing: starting a command at a time. Every field in a crontab line is about time. There is no field for "only after load succeeds".

So the jobs around cron grow their own fixes:

A backfill means running a job again for past days, for example every day of March. cron only knows "now", so it cannot do this.

cron is still right for simple jobs. Three jobs that do not depend on each other need cron and an alert, not Airflow.

In Airflow, you write the order of the steps once, as code. This is the shop job as a DAG, a set of tasks with an order:

import pendulum
from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator

with DAG(
    dag_id="shop_orders_daily",
    schedule="0 0 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
) as dag:
    load_orders = BashOperator(task_id="load_orders",
                               bash_command="python load_orders.py", retries=3)
    build_daily_revenue = BashOperator(task_id="build_daily_revenue",
                                       bash_command="python build_daily_revenue.py")
    refresh_dashboard = BashOperator(task_id="refresh_dashboard",
                                     bash_command="python refresh_dashboard.py")

    load_orders >> build_daily_revenue >> refresh_dashboard

From that one line of order, you get four things:

DAG means directed acyclic graph. Acyclic means it has no loops: a task can never wait for itself. Chapter 3 explains why that matters.

Maxime Beauchemin started Airflow at Airbnb in October 2014, as open source. It joined the Apache Software Foundation in 2016, and became a top-level Apache project in January 2019.

Its key choice is that a pipeline is Python code, not a settings file. So you can review it, test it and keep it in Git like other code.

Two releases changed how you write Airflow:

A provider package adds operators for one outside system, such as AWS or Snowflake. Chapter 20, Managed Airflow, Airflow 3 & Migration, covers the changes in Airflow 3.

Airflow decides what runs and when. It should not do the heavy data work itself. The warehouse, Spark or a container does that work, and Airflow starts it and waits for the result.

People call this the control plane and the data plane. The control plane decides and records. The data plane stores and changes the data.

So build_daily_revenue should send a SQL query to the warehouse. It should not load all the orders into the Airflow worker's memory.

Airflow is built for batch work: runs that start, do a fixed amount of work, and end. It fits jobs that run every hour or every day.

How fresh the data must be decides the tool:

A DAG that runs every minute is a sign that the need has outgrown batch. The work of starting each run fills most of the minute.

Airflow is the most common choice, but not the only one:

Start from the need: how fresh the data must be, how many dependencies there are, and what your team already runs. A team that already runs Airflow well has a strong reason to stay.

Managed services, such as Amazon MWAA, Google Cloud Composer and Astronomer, run Airflow for you, for a price (chapter 20).

cron asks "what time is it?". An orchestrator asks "what is ready to run now?".

1. cron runs on time and cannot say that one job needs another.
2. The worst failure is a quiet success on the wrong data.
3. An orchestrator gives order, recovery, a record of runs, and runs for past days.
4. Airflow decides and records; other systems do the heavy data work.
5. Airflow fits batch work, not work that must react within seconds.

Check yourself: on the bad night, why did nobody get an alert? (Answer: every job ended with success, even the one that read old data.)