APACHE AIRFLOWScale & Production

Concurrency, Pools & Deferrable Operators

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Four gates in series. A task passes all four or it waits. Widening a gate that was not the narrow one changes nothing.

⏱ 4 min · Topic 1 of 7

Sixty tasks are ready. How many actually run? The answer is not your worker count — it is the smallest of four separate limits, and most teams tune the wrong one for weeks.

Set the controls below and watch which bar shortens first. That bar is the binding limit, and it is the only one worth changing.

Sixty tasks are ready. How many actually run?
parallelism32
max_active_tasks_per_dag16
max_active_runs3
pool slots5
Tasks ready to run60
max_active_runs60
max_active_tasks_per_dag16
pool slots5
parallelism5
5 of 60 tasks run — the binding limit is pool slotsEvery limit applies, and the smallest one wins. Raising any of the others changes nothing while pool slots is the constraint — which is why “we increased parallelism and it made no difference” is such a common report.
Change one control at a time and watch which bar shortens first. That bar is the only one worth tuning.
Core mental model

Four gates in series. A task passes all four or it waits. Widening a gate that was not the narrow one changes nothing.

Why it matters

“We increased parallelism and nothing changed” is one of the most common Airflow reports, and it is always the same cause: parallelism was not the binding limit. Every limit applies simultaneously and the smallest wins, so raising a non-binding one has exactly zero effect.

parallelism
The maximum tasks running across the entire installation.
max_active_tasks_per_dag
Per-DAG cap, default 16. Formerly dag_concurrency. Stops one DAG monopolising the fleet.
max_active_runs
How many runs of one DAG may be active at once. The catchup and backfill guardrail from Chapter 12.
Pool
A named limit shared across DAGs, sized to a downstream system rather than to Airflow.
Common mistake

Raising limits one at a time and hoping. You change settings for a week with no effect, because only one limit binds and you have not identified it. Find the binding limit first; everything else is noise.

Better habit

Identify the binding limit before changing anything. The others are irrelevant until it moves.

Remember max_active_tasks_per_dag defaults to 16 — it is often the surprise constraint on a wide DAG.

Size pools to the downstream system, never to Airflow’s capacity.

The smallest limit always wins

Four gates in series. This one sentence explains almost every Airflow throughput question anyone has ever asked.

Remember this

Find the binding limit, change that, and ignore the rest until it moves.