APACHE AIRFLOWAuthoring DAGs

Dynamic DAGs & Dynamic Task Mapping

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Hard
01 · Orientation

What You’ll Master Here

Known at parse time and each needs its own schedule → generate DAGs. Only known at run time → map tasks. Known at parse time and it is three things → just write three tasks.

⏱ 4 min · Topic 1 of 8

Two techniques share this chapter and they are constantly confused, because both are called “dynamic”. Generating DAGs builds many pipelines from config at parse time. Task mapping builds many task instances from data at run time. They solve different problems and they compose.

Work through the scenarios below before reading on. The dividing line is one question: when is the number known?

Two different tools. Which problem do you actually have?
→ Dynamic task mappingOne DAG, one schedule, but N parallel task instances where N is only known when the run starts. expand() creates them at run time from an upstream result.
The dividing line is when the number is known. Known at parse time and you want separate schedules → generate DAGs. Only known at run time → map tasks.
Core mental model

Known at parse time and each needs its own schedule → generate DAGs. Only known at run time → map tasks. Known at parse time and it is three things → just write three tasks.

Why it matters

Reaching for the wrong one produces a pipeline that cannot express what you need. Generating DAGs for a count that only exists at run time means re-parsing to discover work, which cannot happen. Mapping tasks when each item needs its own schedule means one shared schedule for things that should be independent.

Dynamic DAG generation
Building several DAG objects from config in one Python file, registered by assigning into module globals.
Dynamic task mapping
Creating N task instances at run time from a list, using .expand(). The count can come from an upstream task.
Mapped task instance
One instance of a mapped task, identified by its map index. All of them share one task definition.
Common mistake

Using dynamic generation to avoid writing three explicit tasks. You trade three greppable, reviewable tasks for a factory nobody can search for. When something breaks, the failing task_id does not exist anywhere in the codebase — it was computed.

Better habit

Ask when the count is known before choosing a technique. That single question decides it.

Prefer explicit tasks until the count genuinely varies. Dynamic is a cost you pay in readability.

Tag generated DAGs as generated, so nobody wastes twenty minutes looking for their source file.

Parse time versus run time

This chapter is the same distinction Chapter 3 drew, applied to structure. Generation happens while the file is parsed; mapping happens while the run executes.

Remember this

Generate DAGs when the count is known and each needs its own schedule. Map tasks when the count is only known at run time.