What You'll Master Here
it does not play the instruments (the tasks), it decides who plays when and ensures they start only when the players before them have finished.
So far you have learned the individual stages, extract, transform, load. But a real pipeline is dozens of these tasks that must run in the right order, at the right time, and recover when they fail. Orchestration is the conductor that makes that happen. Without it, you have a pile of scripts; with it, you have a pipeline.
This chapter explains why orchestration exists, the central abstraction it uses (the DAG, a directed acyclic graph of tasks and dependencies), how scheduling works (time-based versus event-based triggers), and how orchestrators handle the messy realities of retries, backfills, and catchup.
By the end you will be able to model any pipeline as a DAG, choose how it should be triggered, and reason about what happens when a task fails at 3 a.m. You will also know what tools like Airflow, Dagster, and Prefect actually do, and why Airbnb built Airflow to solve exactly this problem.
Orchestration is the conductor of an orchestra: it does not play the instruments (the tasks), it decides who plays when and ensures they start only when the players before them have finished.
Orchestration is what turns isolated jobs into a dependable system. It is also one of the two guardrails (with observability) that beginners skip and seniors insist on, because it is what makes a pipeline run in the right order and recover when reality intrudes.
- orchestration
- Coordinating pipeline tasks: their order, dependencies, timing, retries, and recovery.
- DAG
- A directed acyclic graph: tasks (nodes) with dependencies (edges) and no cycles.
- task
- One unit of work in a pipeline (e.g. extract orders, transform revenue).
- scheduler
- The component that triggers DAGs by time or by an event.
Wiring tasks together with ad-hoc scripts and cron entries. Order, dependencies, and retries become invisible and fragile; one failure cascades unpredictably.
Model every pipeline as a DAG of tasks and dependencies.
Let the orchestrator enforce order and handle retries, not glue code.
Decide triggering (time vs event) deliberately.
Orchestration separates "what each task does" from "when and in what order tasks run". That separation is what makes a pipeline of many tasks understandable, recoverable, and trustworthy.
Learn the DAG first, then scheduling, then retries and backfills. The DAG is the mental model the whole field uses, get it solid before the rest.
Orchestration is the conductor that runs pipeline tasks in the right order, at the right time, with recovery; it turns a pile of scripts into a dependable system.
- Define a DAG in one sentence.
- Explain why cron entries alone are not orchestration.