APACHE AIRFLOWScale & Production

Testing DAGs & CI/CD Deployment

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Test the wiring separately from the logic. The wiring needs Airflow; the logic should not.

⏱ 4 min · Topic 1 of 7

Airflow has a testing problem that is really a design problem: a DAG file that mixes wiring, configuration and business logic is hard to test at any level. Split those three and each becomes testable with ordinary tools.

Five layers, each catching something the others cannot. Explore them below and note where the cost jumps.

Five layers — each catches something the others cannot
DAG integrity
  • Catches: Import errors, cycles, duplicate dag_ids, missing owners or tags, unknown task ids in a branch.
  • Misses: Anything about whether your logic is correct.
  • Worth it because: The highest value per second in the whole pyramid. It catches the class of bug that takes a DAG off the UI entirely, and it runs on every commit.
Core mental model

Test the wiring separately from the logic. The wiring needs Airflow; the logic should not.

Why it matters

The failure this prevents is specific and common: a DAG with an import error disappears from Airflow entirely. There is no run, no failure and no alert — the pipeline simply stops existing. A twenty-second integrity test on every commit removes that class of incident permanently.

DagBag
The object that parses a folder of DAG files exactly as the dag processor would. What makes integrity tests a fast unit test.
dag.test()
Runs a whole DAG for one logical date in a single process. The best value for verifying the graph.
airflow tasks test
Runs one task immediately with no DAG run created. The fastest development loop (Chapter 4).
Common mistake

Having no automated check that DAGs still parse. A broken import silently removes the DAG from Airflow. Nobody is alerted because nothing failed, and the absence is discovered when someone asks why yesterday’s data is missing.

Better habit

Run the integrity test on every commit. It is the cheapest insurance in this chapter.

Keep business logic in plain functions so it can be tested without Airflow at all.

Reserve the expensive layers for the pipelines that would hurt most if they broke.

Absence is the failure mode

Most systems fail loudly. An unparseable DAG fails by not existing, which no runtime alert can catch. That is why parse-time testing matters more here than in most codebases.

Remember this

Five layers, and the cheapest one catches the failure that alerting cannot.