APACHE AIRFLOWAuthoring DAGs

Branching, Trigger Rules & Task Groups

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

A branch marks the paths not taken as skipped. Skipped is contagious under the default rule, so the fix belongs on the task where the paths reconverge.

⏱ 4 min · Topic 1 of 8

Branching is where a DAG stops being a fixed sequence and starts making decisions. Every task still exists in the graph; only some of them run. That sounds simple, and it produces the single most common “my pipeline silently did nothing” bug in Airflow.

The cause is that a skipped task is not a neutral state — it propagates. Set the branch and the join’s trigger rule below and watch the pipeline work or quietly stop.

A branch, two paths, and one join that has to run either way
The branch chooses
publish trigger_rule
choose_routesuccessfull_reloadskippedincrementalsuccesspublishskipped
publish is SKIPPED — the pipeline silently stops hereThis is the trap. Skipped states cascade through all_success, so the one path the branch did not choose skips publish too. The run goes green having published nothing, and no task failed — there is nothing to alert on.
@task.branch
def choose_route(rows: int) -> str:
    return "full_reload" if rows > 10_000 else "incremental"

@task(trigger_rule="all_success")
def publish() -> None:
    ...

route = choose_route(count())
route >> [full_reload(), incremental()] >> publish()
Core mental model

A branch marks the paths not taken as skipped. Skipped is contagious under the default rule, so the fix belongs on the task where the paths reconverge.

Why it matters

A branch with the default trigger rule downstream produces a green run that published nothing. No task failed, so no alert fired, and the Grid view is a wall of pale skipped squares that look like they were meant to be there. This is the failure this whole chapter exists to prevent.

@task.branch
Returns the task_id or ids to follow. Every other downstream path is skipped. Returning None skips all of them.
trigger_rule
The condition a task requires of its upstream tasks before it will run. Defaults to all_success.
Skip propagation
Skipped tasks cascade through the all_success and all_failed rules, causing those tasks to skip as well.
Common mistake

Adding a branch and leaving the join task on the default trigger rule. The unchosen path skips, the skip cascades into the join, and everything below it skips too. The run finishes green having produced nothing, and there is nothing to alert on because no task failed.

Better habit

Every time you add a branch, immediately set the trigger rule on the task where the paths rejoin.

Look for pale squares in the Grid view after a branch. Skipped is a state worth reading, not scenery.

Prefer none_failed_min_one_success on a join — it tolerates the skipped sibling but not a wholly empty branch.

Skipped is not neutral

It is a state that spreads. Under the default rule a single skip near the top of a DAG can skip everything below it, and the run still reports success.

Remember this

Branching is easy; rejoining is the part that needs thought. The trigger rule on the join is the whole difference.