Orchestration & Scheduling

Last week you learned what one job does. This week asks a different question. Who runs the job, in what order, and what happens when a step fails?

What you already know

• A pipeline has five stages. A job is the code that runs them, and a run is one time it runs.
• ETL changes data before the load. ELT loads first, then changes it in the warehouse.
• A load can add rows, replace a day, or merge changes.
• A job you run twice must give the same result. This is idempotency.
• An incremental job reads one window of dates. A backfill runs it for a past day.

What you will learn

1. What an orchestrator adds to a plain cron line.
2. A DAG: tasks and the arrows between them.
3. Waiting for a file instead of guessing a time.
4. Retries and timeouts for each task.
5. What happens when one task fails.
6. What to alert on, and how to pick a tool.

We use the shop job from week 7. It copies orders into the warehouse and builds a daily_revenue table. Now we split it into five tasks.

A task is one step of the job.

Cron is a Linux tool that starts a command at a set time. Here is the shop job as one cron line:

0 2 * * *  python daily_revenue.py

Cron does this one thing well. It does not know what comes before or after your command. An orchestrator is a tool that runs tasks in the right order and watches them.

It does four things:

1. Schedule: it starts the job at the right time.
2. Order: a task starts only after the tasks it needs have finished.
3. Retries: it runs a failed task again, up to a limit.
4. Alerts: it tells a person when a task still fails.

Cron gives you only the first one. Order is the one that hurts most. Say you split the job into two cron lines, and give the second a gap of 30 minutes:

0 2 * * *   python extract_orders.py
30 2 * * *  python build_daily_revenue.py

A task is one step of the job. A dependency is a rule that says task B needs task A to finish first. In a picture, it is an arrow.

A graph is boxes joined by arrows. All the tasks and arrows together make a DAG. Look at the five tasks of the shop job, and follow the arrows from left to right:

DAG means directed acyclic graph. Directed: every arrow has one direction. Acyclic: you can never follow arrows and come back to where you started.

Here is one run of this DAG. Its logical date is 2026-01-21, the day of data it works on (week 7). Only order 106 has that date:

Why no cycles? Suppose A needs B and B needs A. Neither can start first, so an orchestrator refuses a cycle.

Our shop job is one line of tasks. A bigger job can branch.

Say a build_daily_refunds task also needs load_raw_orders, but not build_daily_revenue. There is no arrow between the two builds, so they can run at the same time.

A schedule starts a run at a time. But the data may not be ready at that time.

Now say the orders come as a file instead. Another team sends orders_2026-01-21.csv at about 01:00.

Some nights the file is late. A job that starts at 02:00 would then read nothing, or half a file.

A sensor is a task that waits until something is true, such as a file existing. It checks again and again, and then lets the next task start. Here is a sensor in front of the job, in simple made-up code:

task wait_for_file
  look for  orders_2026-01-21.csv
  check     every 5 minutes
  timeout   3 hours

task extract_orders
  needs     wait_for_file

A timeout is the longest time a task may run. Then it fails. Without a timeout, a file that never comes keeps the sensor waiting for a very long time.

The other way is an event trigger: the arrival of the file itself starts the run. Both ways replace the guess "it usually arrives by 02:00" with a fact.

This run starts on 2026-01-22, but its logical date is 2026-01-21. The Airflow chapter on logical dates shows how Airflow sets it.

Many failures go away by themselves, such as a network drop. A retry runs the failed task again after a short wait. A timeout stops a task that runs too long.

You set both for each task, because tasks fail in different ways. Here are example values for the shop job:

A retry only works if running the task twice is safe. That is idempotency. load_raw_orders replaces the whole day, so a second run leaves the same rows.

Do not retry a check that failed on the data. The rows do not change because you looked again, so the retry fails too. It only makes the alert come later.

A backfill, which you met in week 7, is not a retry. A retry repeats a failed task in one run. A backfill is a new run for a past logical date.

By default, if a task fails, the tasks that need it do not run. In this chapter we call them skipped.

Say the load fails in the run for 2026-01-21, and every retry fails too. Look at the same five tasks again:

The run now has one failed task and three skipped tasks. Here is what each part holds:

This is the good result. The dashboard shows data that is old but right.

Without the skip, the build would run on empty data. The dashboard could show 2026-01-21 with no revenue.

To fix it, you repair the cause and run again from the failed task. The tasks that already succeeded do not need to run again.

Airflow calls this state upstream_failed. Upstream means before it in the DAG. Airflow keeps the word skipped for tasks left out on purpose.

An alert is a message that tells a person to act. A good alert is rare and clear. If people get alerts all day, they stop reading them.

Here is what the shop job should and should not alert on:

The last row is a late run without any failure. Alert on a missing result as well as on an error. A silent job is the worst case.

The message must say four things: the job, the logical date, the failed task, and where to read the log.

The log is the text a task writes while it runs. For example: "Job daily_revenue, date 2026-01-21: load_raw_orders failed after 2 retries. Log: (link)."

Airflow, Dagster and Prefect are orchestrators. dbt builds warehouse tables from SQL files, and it makes its own DAG. Each tool has a different idea of what a task is:

dbt is different, because it works only inside the warehouse. You write one SELECT per table. dbt reads the ref() calls to find the arrows. Here is the daily_revenue model:

-- models/daily_revenue.sql
SELECT order_date,
       SUM(amount) AS revenue
FROM   {{ ref('raw_orders') }}
WHERE  status = 'paid'
GROUP  BY order_date

So dbt draws the DAG for you from the ref() calls. It still needs something to start it each night, such as cron, one of the other tools, or dbt’s own paid scheduler.

One job with no dependencies can stay on cron. Move to an orchestrator when you answer yes to these questions:

1. Do jobs need other jobs to finish first?
2. Do you need retries, timeouts and alerts for each task?
3. Do you need to see the history of every run?

If you answer yes, ask one more question. Who will run the tool: your team, or a company you pay?

Then ask what happens after an outage, a time when the tool is down. If the orchestrator is down for a day, some runs are missed.

Whether it replays them later is a setting. In Airflow it is called catchup. The Airflow chapter on logical dates shows it.

Three scenarios for this week. Build each one on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.