The full concurrency ladder, pools that protect your warehouse, and converting idle sensors into deferrable tasks that free their worker slot.
⏱ 14 min readTopics chapter readerLevel · Medium
01 · Start here
Start Here
⏱ 6 min · Topic 1 of 10
The last chapter chose an executor. It decides how tasks are sent out. It does not decide how many may run at once.
This chapter is about that number. Several limits sit between a ready task and a running task. The smallest one wins.
A task that waits for room stays in state scheduled. A task in state queued already has room. The scheduler gave it to the executor, and it waits for a worker.
You follow shop_orders_daily again: load_orders, then build_daily_revenue. The run is about 2026-01-20, dag_run 380.
You also meet made-up DAGs that share the shop Postgres database. It can take only 4 connections at once. Their dag_run ids start at 401, and every number about them is made up.
This chapter is about Airflow 2. You answer three questions:
Which limits exist, and which one is full?
How do you protect a shared system, and choose who goes first?
How can a task that only waits give its slot back?
02 · The limits
Limits on Running Tasks
⏱ 6 min · Topic 2 of 10
A limit is a number that caps how many tasks run at once. Airflow has five. Each one caps a different group of tasks:
The five limits (Airflow 2 defaults)
Limit
What it caps
Default
Where you set it
parallelism
Tasks running or queued, per scheduler. With one scheduler, that is all of Airflow.
32
The core section of airflow.cfg
max_active_tasks
Tasks running or queued in one DAG, across its runs
16
A DAG argument. The default comes from max_active_tasks_per_dag.
max_active_runs
Runs of one DAG that are active at once
16
A DAG argument. The default comes from max_active_runs_per_dag.
max_active_tis_per_dag
Copies of one task running or queued, across runs
No limit
A task argument
Pool
Tasks that share a set of slots, from any DAG
default_pool has 128 slots
The pool argument of a task
A task instance is one task in one run. That is what these limits count.
Your executor adds one more cap: worker slots. With Celery, each worker has worker_concurrency slots, and the default is 16.
Airflow 3 keeps pools and these limits. Check its release notes for changed names or defaults.
Now see where you set them. The values are made up, and the pool comes in the next section:
Set the limitsworked example
Python
# airflow.cfg
# [core]
# parallelism = 32
with DAG(
dag_id="shop_orders_daily",
max_active_runs=1,
max_active_tasks=8,
# schedule and start_date as before
) as dag:
load = PythonOperator(
task_id="load_orders",
python_callable=load_orders,
max_active_tis_per_dag=1,
pool="shop_db",
)
Set nothing and the defaults from the table apply.
Watch out
A full limit does not fail a task. A full max_active_runs keeps the new run in run state queued. Any other full limit keeps the task in state scheduled.
03 · Protect
Pools Protect Shared Systems
⏱ 11 min · Topic 3 of 10
Some limits protect Airflow. A pool protects something else: a database, an API or a warehouse that can take only so many users at once.
A pool is a named set of slots. A slot is one place for a task. A running or queued task holds its slot until it ends.
A pool is the only limit for a group of tasks you choose, from many DAGs. parallelism counts all tasks, and a per-DAG limit sees only one DAG.
The shop Postgres database takes 4 connections, so make a pool with 4 slots. One command makes it:
Create the pool shop_dbworked example
Shell
airflow pools set shop_db 4 "Shop Postgres connections"
airflow pools list
Result · 2 rows
pool
slots
default_pool
128
shop_db
4
The list is simplified. default_pool exists already. A task with no pool uses it.
The name shop_db is made up. The command takes a name, a number of slots and a description.
Now put tasks in the pool. Each task names it with the pool argument:
Four tasks run and fill the 4 slots. The two backfill tasks are ready, but there is no room, so they stay scheduled.
The runs are dag_run 380 and 401 to 405. All are made up except 380.
The next section explains why these four go first. Now watch the pool over time:
The pool shop_db over time (made up)
Time
Slots used
Slots free
Waiting in scheduled
00:00:00, before the scheduler hands out slots
0
4
All 6 tasks
00:00:30
4
0
backfill_2025_01 and backfill_2025_02
00:01:00, four tasks end
2
2
None
00:02:00, all tasks end
0
4
None
The two backfill tasks waited 1 minute in scheduled. They did not fail.
A heavy task can take more than one slot with pool_slots=2.
A task that needs more slots than the whole pool has cannot start. It stays in scheduled. Only the scheduler log shows a warning.
Watch out
Size the pool from the shared system, not from Airflow. Airflow could run 32 tasks here, the parallelism default. The shop database takes 4, so the pool has 4 slots.
04 · Combine
The Smallest Limit Wins
⏱ 7 min · Topic 4 of 10
Every limit applies at the same time. A task runs only if it passes all of them. So the number of running tasks is the smallest limit.
Take a made-up DAG, shop_export_regions, dag_run 406. It has 40 export tasks, all ready together. The workers have 64 slots, as in the last chapter.
Look at five setups and how many tasks run in each:
5. Keep rows 3 and 4. Pretend the pool has 20 slots
20
20
Pool shop_db
Waiting is 40 minus the tasks that run. The shop database takes only 4 connections, so row 5 is not safe.
Check row 1. The limits are 32, 64, 128 and 16, so 16 tasks run.
Row 2 adds a pool of 4. Now 4 is the smallest, so 4 run.
Rows 3 and 4 raise limits that were not full. Nothing changes: 4 still run and 36 still wait. This is why "I raised parallelism and nothing happened" is a common report.
Raise only the limit that is full. Stuck in Scheduled or Queued shows how to find it.
Watch out
max_active_runs counts runs, not tasks. With max_active_runs=1, a second run waits in run state queued. None of its tasks start, even if every task limit has room.
05 · Order
Who Goes First
⏱ 9 min · Topic 5 of 10
When slots are short, the scheduler needs an order. It starts the task with the highest priority_weight first. The default weight is 1.
Priority only orders tasks that compete for the same slots. It never adds a slot.
The shop_db pool has 4 slots and 6 ready tasks. Production tasks get weight 10. The two backfill tasks keep weight 1.
The reason is real. With equal weights, Airflow generally prefers the older run. Backfill runs are old, so they could take slots from today.
You can set a weight on every task of a DAG with default_args:
Give a DAG a higher weightworked example
Python
with DAG(
dag_id="shop_orders_daily",
default_args={"priority_weight": 10},
# schedule and start_date as before
) as dag:
load = PythonOperator(task_id="load_orders", python_callable=load_orders, pool="shop_db")
build = PythonOperator(task_id="build_daily_revenue", python_callable=build_daily_revenue)
load >> build
Both tasks get weight 10. The other made-up DAGs set weight 10 the same way. The backfill DAG sets nothing.
The scheduler compares the effective weight, not only your number. The default rule is downstream: a task adds its own weight plus the weights of all tasks after it.
Look at the six tasks in the pool at 00:00:00:
Effective weights, downstream rule (made up)
Task
Own weight
Tasks after it
Effective weight
Gets a slot?
load_orders
10
build_daily_revenue (10)
20
Yes
pull_refunds
10
None
10
Yes
pull_customers
10
None
10
Yes
export_clicks
10
None
10
Yes
backfill_2025_01
1
None
1
No. It waits in scheduled.
backfill_2025_02
1
None
1
No. It waits in scheduled.
The top four are 20, 10, 10 and 10. Ties do not matter here, because all four winners get a slot.
This matches the last section: the two backfill tasks wait. There are two other rules. Look at all three for our DAG, with weight 10 on each task:
weight_rule for load_orders >> build_daily_revenue
weight_rule
load_orders
build_daily_revenue
Effect
downstream (default)
20
10
Tasks with more work after them go first
upstream
10
20
Tasks later in the DAG go first, so started runs finish
absolute
10
10
Only your own number counts
Upstream adds the weights of the tasks before. build_daily_revenue has load_orders before it.
Watch out
Priority does not jump a full pool. A task with weight 20 still waits while all 4 slots are used. It only goes first when a slot frees.
Keep high weights rare
If every task has a high weight, none is higher than another. Give a high weight only to the few tasks that must go first.
06 · Waiting
Sensors Hold Slots While Waiting
⏱ 8 min · Topic 6 of 10
A sensor is a task that waits for something, such as a file. It checks again and again until the thing arrives.
Take a made-up DAG, supplier_stock_daily, dag_run 407. Its task wait_for_file waits for a file that arrives about 03:00. Say 40 DAGs have such a sensor.
In poke mode, the default, a sensor keeps its worker slot for the whole wait. It sleeps between checks, but the slot stays taken.
Parallelism is 32. So 32 sensors run and take all 32 slots for 3 hours. The other 8 wait in scheduled, and no real task can start.
Reschedule mode is the first fix. Between checks the sensor stops and leaves its slot. Look at the code and at one sensor over time:
Each check takes about 2 seconds, made up. Between checks the slot is free. By default a pool counts only running and queued tasks, so a sleeping sensor holds no pool slot.
S3KeySensor needs the Amazon provider package. The bucket name is made up. The interval is 300 seconds, and the timeout is 6 hours.
Reschedule has a cost. Every check goes through the scheduler again. It adds database work and a small delay, so use intervals of minutes, not seconds.
A third way, deferrable, hands the wait to the triggerer. The triggerer is a part of Airflow that waits for events. The next section explains it.
Look at the three ways to wait:
One sensor, three ways
poke (default)
reschedule
deferrable
State while waiting
running
up_for_reschedule
deferred
Holds a worker slot?
Yes, for the whole wait
Only during each check
No
Who waits?
The worker
The scheduler starts each check
The triggerer
40 sensors, parallelism 32
32 run, 8 wait in scheduled
Slots are free between checks
No slots used while waiting
Watch out
The timeout covers the whole wait, not each check. In reschedule mode it counts from the first check. When it ends, the sensor fails.
07 · Give the slot back
Deferrable Tasks and the Triggerer
⏱ 9 min · Topic 7 of 10
A deferrable task gives its worker slot back while it waits. The triggerer waits for it, together with many other tasks.
The triggerer runs small pieces of code called triggers. A trigger watches for one event, such as a new file. Many triggers share one process.
When the event happens, the trigger fires. The task goes back to the scheduler and finishes on a worker.
Many provider operators accept deferrable=True. Look at the same sensor, now deferrable:
The times are made up. The task is in state deferred between 00:00:05 and 03:00:11.
This needs Amazon provider 8.2.0 or newer. Some older operators have a separate class with Async in its name.
Count the slot time. The task held a worker slot from 00:00:04 to 00:00:05, and from 03:00:14 to 03:00:15. That is 2 seconds.
In poke mode, the same wait holds one slot from 00:00:04 to 03:00:15. That is 3 hours and 11 seconds. With 40 such sensors, all 32 slots stay free for real tasks.
By default a deferred task holds no pool slot. A pool setting, include_deferred, can count it. It exists in newer Airflow 2 releases, so check your version.
A deferred task can still time out. If the trigger does not fire in time, Airflow resumes the task with an error, and it fails. Set a timeout you can accept, because the sensor default is 7 days.
Defer only long waits. Each defer adds one more trip through the scheduler and a new start on a worker, as the timeline shows.
Watch out
With no triggerer running, a deferred task waits until its timeout, then fails. With the default timeout, that is a week of silence. Start it with airflow triggerer, and alert on its heartbeat, a regular I am alive signal.
Keep triggers fast
A slow call inside a trigger blocks the other triggers on that process. Use the built-in deferrable operators of providers when you can.
08 · On call
Stuck in Scheduled or Queued
⏱ 8 min · Topic 8 of 10
When a task waits, read its state first. The state tells you which part holds it. Look at the states and where to look:
State
Meaning
Look at
none
The upstream tasks are not done.
The upstream tasks
scheduled
Ready, but no room yet.
The pool and the limits
queued
The executor has it. It waits for a worker.
Workers, queue names and the broker
up_for_reschedule
A sensor sleeps between checks.
Nothing. The next check comes.
deferred
The triggerer waits for an event.
The triggerer
A dag_run that stays in state queued waits for max_active_runs.
For scheduled, check the limits from narrowest to widest. Stop at the first one that is full:
Does the pool exist? A task that names a missing pool never leaves scheduled. The scheduler log warns about it.
Is the pool full? Used slots equal total slots.
Is max_active_tis_per_dag full? Running and queued copies of this task equal the limit.
Is max_active_tasks full? Running and queued tasks of this DAG equal the limit.
Is parallelism full? Running and queued tasks of all DAGs equal the limit.
Try it. At 00:00:30, backfill_2025_02 is in scheduled. Check the pool first, with the made-up rows of the shop_db example:
The pool has 4 slots, so it is full. That is the cause. The scheduled rows are not counted.
The Pools page in the Airflow UI shows the same numbers.
You stopped at step 2. The fix is in the pool: wait, or find what holds the slots. Raising parallelism or the DAG limits changes nothing.
Also alert on tasks that stay in scheduled or queued longer than usual. One alert catches every cause in this chapter.
Watch out
Do not look upstream for a scheduled or queued task. Its upstream tasks already passed. Looking there wastes the first minutes of an incident.
Interview question
Why is a task stuck in scheduled? A strong answer says the upstream tasks passed. Then it checks the pool, the per-DAG limits and parallelism, in that order.
09 · Recap
Quick Recap
⏱ 5 min · Topic 9 of 10
Several limits stand between a ready task and a running task, and the smallest one decides. A pool protects a shared system. A deferred task gives its slot back.
Look at the answers you should know:
Question
Short answer
Which state shows a full pool?
scheduled. The task is ready, but there is no room.
What does queued mean?
The executor has the task. It waits for a worker.
Defaults of parallelism, max_active_tasks, max_active_runs?
32, 16 and 16.
Which limit caps a group of tasks you choose, from many DAGs?
A pool.
What does priority_weight do?
It orders tasks that compete for slots. It adds no slot.
What is the default weight_rule?
downstream: own weight plus the weights of all later tasks.
What does a deferred task hold?
No worker slot. The triggerer waits for it.
What if no triggerer runs?
Deferred tasks wait until their timeout, then fail.
Try this alone. 8 tasks are ready in a pool of 3 slots. Each takes 1 minute.
How long until all finish? Which state do the waiting tasks have?
The answer is 3 minutes: waves of 3, 3 and 2 tasks. The 5 tasks that wait at first are in scheduled.
10 · Next Chapter
Next Chapter
You now know how many tasks may run, and how a waiting task gives its slot back. The next chapter connects DAGs to each other.
⏱ 3 min · Topic 10 of 10
Next chapter
Assets & Cross-DAG Data-Aware Scheduling
You now know how many tasks may run, and how a waiting task gives its slot back. The next chapter connects DAGs to each other.
A downstream DAG can wait for data to appear, not for a clock. It also covers the Airflow 3 rename of Dataset to Asset.
