APACHE AIRFLOWAuthoring DAGs

Dependencies & the TaskFlow API

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

A dependency is an edge. Everything in this chapter is a different notation for drawing the same edges — pick the notation that makes the shape obvious to a reviewer.

⏱ 4 min · Topic 1 of 9

Two things share this chapter, and they are the two halves of authoring a real DAG: how you write a task, and how you wire tasks into a shape. Most pipelines are not a straight line — they fan out, fan in, and cross — and the shapes you can express determine the pipelines you can build.

Start with the shapes. Pick one below and read the graph, then read every syntax that produces it. The Cross shape is worth lingering on, because it is the one that cannot be written the obvious way.

Pick a dependency shape — see the graph and every syntax that builds it
The graph
extractcleanvalidateload
extract → clean, validate → loadThe most common real shape: split the work, do the halves in parallel, then join. Both middle tasks must finish before load starts.
extract >> [clean, validate] >> load

# one line, four tasks, two levels of parallelism
Core mental model

A dependency is an edge. Everything in this chapter is a different notation for drawing the same edges — pick the notation that makes the shape obvious to a reviewer.

Why it matters

Dependency wiring is where a DAG stops being three tasks in a row and starts being a pipeline. It is also where a quiet class of bug lives: a mis-written fan-out gives you fewer edges than you intended, the DAG still parses, the Graph view still looks plausible, and tasks run in an order you did not design.

>>
The shift operator, read as “must finish before”. a >> b means b waits for a.
Fan out
One task followed by several that can run in parallel.
Fan in
Several tasks joining into one that waits for all of them.
chain()
A helper that wires a sequence of tasks and lists in order. Pairs equal-length lists element-wise.
cross_downstream()
A helper connecting every task in one list to every task in another — the full cross that lists alone cannot express.
Common mistake

Assuming [a, b] >> [c, d] connects everything to everything. It does not build the four edges you pictured. Use cross_downstream() when you genuinely need every upstream connected to every downstream — otherwise you get a graph with missing edges and tasks that start earlier than intended.

Better habit

Draw the shape before writing the code. Almost every wiring bug is a shape that was never drawn.

Prefer the notation that makes the shape visible: lists for hand-written graphs, chain() for generated ones.

Check the Graph view after any wiring change. It is the fastest confirmation that the edges are what you meant.

Shape first, syntax second

Every notation here produces identical graphs. The only question is which one makes your intent obvious six months later.

Remember this

Learn the five shapes and the notations that build them. Everything else in DAG authoring is filling them in.