Airflow Scaling & Concurrency

The reasoning

From the outside in. `core.parallelism` caps concurrently running task instances across the whole deployment — the global ceiling, and the one people forget exists until it is binding. `core.max_active_tasks_per_dag` (historically `dag_concurrency`) caps running tasks within one DAG, and is overridable per DAG with `max_active_tasks`.

`max_active_runs` caps concurrent dag runs of one DAG. It is the one that makes a DAG look idle during a catchup: runs exist, they are queued, and only N execute. `max_active_tis_per_dag` on a task caps instances of that particular task — the lever for a mapped task that would otherwise fan out to a thousand.

Pools cut across all of it. A pool has a fixed number of slots, a task can request more than one with `pool_slots`, and a task in a full pool waits regardless of how much cluster capacity is free. That is what makes pools the right tool for protecting an external system — a database that tolerates five concurrent connections gets a five-slot pool, and nothing else in the cluster can change that.

The practical consequence: a task can be blocked by any of the five, and the UI does not tell you which. The pool page shows slot usage, the DAG's own settings are in the code, and `parallelism` is in the config. Checking them in that order — pool, then DAG, then global — resolves most stalls, and the answer is almost never 'add more workers', because in four of the five cases the workers are idle.

The reasoning

A pool is a named set of slots that tasks queue for. `pool='legacy_db'` with five slots means at most five such tasks run at once regardless of which DAG they belong to, how many workers are free, or what any DAG's own concurrency says. That is exactly the shape of the requirement — the constraint belongs to the database, not to any one pipeline.

The alternative — asking each team to set `max_active_tasks` — fails because the limits do not compose. Three DAGs each limited to two can still put six connections on the database, and a fourth DAG written next quarter knows nothing about the rule. A pool is the only mechanism where the total is enforced.

`pool_slots` lets a heavy task claim more than one slot, which is how you express 'this one query is worth three normal ones'. And tasks not assigned a pool go to `default_pool`, whose size is another global limit worth knowing about — shrinking it is a blunt way to throttle everything.

Two operational notes. Tasks waiting on a pool sit in `queued`, so a full pool looks exactly like a capacity problem unless you check the pool page — that is the single most useful screen for this class of stall. And a task that holds a pool slot while doing nothing, such as a sensor in `poke` mode, is the classic way a pool intended to protect a database ends up starving the cluster instead.

The reasoning

When slots free up, the scheduler picks queued task instances by priority within each pool. `priority_weight` defaults to 1, and `weight_rule` decides how it combines with the graph: `downstream` (the default) sums the task's own weight with all its descendants, `upstream` does the reverse, and `absolute` uses just the number you set.

The default has a consequence worth stating: a task at the head of a twenty-task chain has an effective weight of twenty-one, while a leaf has one. So long pipelines naturally start before short ones, which is usually sensible — it gets work moving — and is definitely surprising the first time you set explicit weights and they do not behave as written. `weight_rule='absolute'` is what you use when you want the number to mean what it says.

To make finance jump the queue: raise `priority_weight` on its tasks, via `default_args` so the whole DAG is consistent. But priority only orders tasks *within a pool*, so if finance shares a pool with everything else, a high weight helps and does not guarantee. The stronger guarantee is a dedicated pool with reserved slots, so finance always has capacity that nothing else can take.

The trade-off to name: reserved capacity is idle when finance is not running, and priority is free but only relative. For a genuine business deadline I would use both — a small reserved pool for the critical path, and priority within the shared pool for everything else. And I would be sceptical of a scheme where everything is high priority, which is the usual end state and is the same as no priorities at all.

The reasoning

`max_active_tis_per_dag` on the mapped task caps how many of its instances run simultaneously within a DAG; `max_active_tis_per_dagrun` scopes that to a single run. Eight instances are created either way and all eight execute; the cap only staggers them. The published task instances show all eight succeeding, which is the point the question is making.

The pool is the stronger and more shareable control, for the same reason as any external limit: the vendor's constraint is two concurrent calls *in total*, and if a second DAG also calls that API, a per-task cap on one of them enforces nothing. A two-slot pool named for the API is the constraint that holds across everything.

What actually limits the fan-out width is upstream: the list the task expands over. If you need to cap that, cap it in the producing task, or the number of instances is whatever the source happened to return that day. `max_map_length` is the hard backstop — the point at which Airflow refuses rather than trying to schedule a hundred thousand instances — and it is a safety net, not a design.

The failure to plan for: a mapped task with no cap fanning out unexpectedly wide starves every other DAG, because those instances count individually against `parallelism` and the pool. The combination worth defaulting to on any mapping over external data is a cap on the task, a pool for the resource, and a sanity check on the list length in the upstream task.

The reasoning

In `poke` mode a sensor is a running task for its whole wait. Two hundred sensors against forty-eight slots means the cluster is fully occupied by tasks that are doing nothing but sleeping, and the remaining hundred and fifty-two sensors queue behind them. It is a deadlock in everything but name: no work can start, and the sensors that are running will hold their slots for hours.

The one-line mitigation is `mode='reschedule'`: the sensor exits after each check and is rescheduled after `poke_interval`, freeing the slot in between. It costs scheduler churn and gives you poke-interval granularity, and for two hundred long waits it is a transformation.

The real fix at this scale is deferrable operators. The task suspends into the `triggerer`, a separate process running thousands of lightweight async waits in one event loop, and is resumed when the condition fires. No worker slots, second-level responsiveness, and the cost is one more component to deploy. That is the answer for two hundred sensors.

The structural fix is to stop waiting. Two hundred sensors is two hundred pipelines polling for something someone else produces, which is a notification problem solved by polling. Datasets, an event from the producer, or the upstream DAG triggering the downstream removes the wait entirely. Where that is impossible — a genuinely external vendor — a sensor pool caps the damage so the class of problem can never take the whole cluster again, and every sensor gets a `timeout` so a file that never arrives fails instead of waiting forever.

The reasoning

The sequence: the operator does its initial work, then defers with a trigger object and a method name to resume at. The worker slot is released immediately and the task instance goes to `deferred`. The triggerer — a separate component you must deploy — picks up the trigger and runs it as asyncio, alongside thousands of others in one process. When it yields an event, the task is queued again and the named method runs with that event.

That is why it scales: waiting becomes an async coroutine rather than an occupied process. One triggerer handles thousands of concurrent waits on a fraction of the resources one worker uses, and responsiveness is seconds rather than the poke interval.

What you need: the triggerer running (and it should be more than one for redundancy, since triggers are redistributed if one dies), and operators that support it — most sensors and many provider operators take `deferrable=True`, and there are async variants of the common ones. If nothing runs the triggerer, deferred tasks stay deferred forever, which is a confusing outage the first time.

The constraints worth knowing. Triggers must be genuinely async and serialisable — anything blocking inside one stalls the whole event loop, which is the main way people write a bad custom trigger. And the resume method starts fresh, so any state must have been carried in the trigger, not held in memory. Writing a custom deferrable operator is real work; using the provider ones is a config flag, which is why the first move is always to check whether the operator you already use supports it.

The reasoning

The deciding number is startup overhead relative to task duration. A Kubernetes pod takes seconds to schedule, pull and start; on a thirty-second task that is a large fraction of the runtime, and across twenty thousand tasks it is hours of pure overhead plus serious churn on the cluster's control plane. Long-lived Celery workers pay that cost once and then execute tasks immediately.

Flip the durations and the calculus inverts. Twenty-minute tasks make a few seconds of startup irrelevant, and you get per-task images, per-task resource requests, real isolation between tenants, and scale-to-zero when nothing is running. That is a better fit for heterogeneous heavy work.

What Celery costs at that volume: a broker to operate and monitor, workers as a fleet with autoscaling — KEDA on queue depth is the common answer — and the fact that long-lived workers share one Python environment, so two DAGs with conflicting dependencies are a real problem. That last one is the usual reason teams move to pods for a subset of tasks.

Which is why the honest answer for a large estate is hybrid: Celery for the many short orchestration-shaped tasks, and `KubernetesPodOperator` for the heavy or dependency-conflicting ones. You get fast dispatch where it matters and isolation where it matters, and you avoid sizing one fleet for both. Queue routing is what makes that manageable — light tasks to small workers, heavy ones to a queue with big workers or to pods.

The reasoning

With Celery, each worker subscribes to one or more named queues and each task declares which queue it goes to. So you run thirty small workers on `default` and two large ones on `heavy`, and the thirty-two-gigabyte tasks are the only things that reach the large workers. Fleet cost drops to roughly the size of the actual workload rather than the size of its worst member.

The pitfall to name because it produces a mystifying stall: a task routed to a queue no worker is consuming sits in `queued` forever, looking identical to a capacity problem. A typo in a queue name, or a worker deployment that was scaled to zero, produces exactly that. It is worth a check that every queue referenced by a DAG has a consumer.

Queues are about *where* a task runs; pools are about *how many* run at once. They are frequently confused and they compose: a task can be routed to the `heavy` queue and also be in a five-slot pool protecting a database. Being able to state that distinction cleanly is usually what the interviewer is listening for.

The alternative for the heavy tail is not to have big workers at all: send those tasks to a `KubernetesPodOperator` with its own resource requests, and let the cluster schedule them. Then the Airflow fleet is uniformly small and the heavy work is isolated — which is also what stops one memory-hungry task killing a worker that was running twenty other things.

The reasoning

Start with parsing, because it is usually the problem. `airflow dags report` gives the parse time of every file; the worst offenders are files with heavy top-level code or huge dynamically generated DAG sets. Fixing those — moving work into tasks, using templated variable access, generating DAG files in CI rather than at parse time — is the highest-leverage change and often the only one needed.

Then the parsing configuration: `min_file_process_interval` raised if DAGs genuinely change rarely, `parsing_processes` to parallelise the parse across cores, and in Airflow 3 the dag processor running as its own component so parsing cannot starve scheduling. Splitting one enormous DAG file into many also helps, since parsing is per file.

Then horizontal scale. Multiple schedulers are supported since 2.0 and coordinate through row-level locking in the metadata database — genuinely active-active, not failover. Two or three replicas is a normal production setup. But they all hammer the same database, which is why the database is the next bottleneck.

Then the database, which at this scale is usually the real ceiling: connection counts from schedulers, workers and triggerers; write volume from heartbeats; and table bloat from years of task instances that `db clean` has never touched. Tuning `scheduler_heartbeat_sec` and `job_heartbeat_sec` reduces write pressure. The metric to watch throughout is scheduler loop duration and the delay between a task becoming eligible and being queued — if that is flat while parse time is high, your problem is parsing, and if both are high, it is the database.

The reasoning

First, is anything running at all? If workers are idle while tasks are queued, it is a limit or a routing problem, not capacity — and that single observation eliminates 'add more workers', which is where most people start. Check that workers are alive and subscribed to the queue those tasks were routed to; a task sent to a queue with no consumer waits forever.

Second, which limit is binding. Check the pool page for full pools, then the DAG's `max_active_tasks` and `max_active_runs`, then cluster `parallelism`. One of these is usually it, and the fix is a config change rather than infrastructure.

Third, if workers are busy, look at *what* they are busy with. If it is mostly sensors, you have the capacity-consumed-by-waiting problem and the fix is reschedule mode or deferrable operators. If it is genuine work that has become slower, the problem may be downstream — a warehouse under load makes every task take longer and the backlog is a symptom of that rather than of Airflow.

Fourth, the scheduler. If tasks are slow to even reach `queued`, it is scheduler lag, and parse time is the first thing to check. Fifth, and only then, capacity: if everything is genuinely saturated with real work, add workers or spread the load — stagger schedules away from the top of the hour, which is where a surprising amount of self-inflicted contention comes from. What I would not do first is restart the scheduler; it fixes the case where the executor lost track of tasks, and it destroys the evidence for everything else.

The reasoning

`max_active_runs` is about *runs*: how many logical dates of this DAG can be in flight at once. During a catchup, sixty runs are created and only `max_active_runs` of them are active; the rest sit queued. In the UI the DAG looks like it is barely doing anything while sixty runs wait, which is exactly the symptom people report as 'catchup is stuck'.

`max_active_tasks` is about *tasks*: how many task instances of this DAG run simultaneously, across all its active runs. A DAG with a wide fan-out hits this one, and the symptom is different — runs progress, but slowly, with tasks queued inside them.

They interact in a way worth stating: with `max_active_runs=4` and `max_active_tasks=16`, four runs share sixteen task slots, so one slow run can starve the others. And both sit under cluster `parallelism`, so a DAG can be well within its own limits and still blocked globally.

How to choose. `max_active_runs=1` for anything writing to a shared target — it is the cheapest protection against two runs racing, and it is what makes a catchup safe. Higher when runs are genuinely independent, such as per-day partitions that never overlap. `max_active_tasks` is mostly about being a good neighbour: capping it stops one DAG's fan-out consuming the whole cluster, which pools do more precisely but this does with one line.

The reasoning

Expansion happens after the upstream succeeds, so nothing warns you beforehand. Forty thousand task instance rows are written, the scheduler starts working through them, and they count individually against `parallelism` and any shared pool — so every other DAG in the deployment queues behind one task's bad day. The metadata database takes the write volume too, which is what makes the recovery slow.

`max_map_length` is the backstop: Airflow refuses to expand beyond it and fails the task rather than trying. That converts a cluster-wide outage into one failed task, which is the right trade and is why the default exists — worth checking it has not been raised casually.

The layers I would actually have in place. A sanity check in the *upstream* task: if the list is more than a few times the normal size, raise, because a forty-thousand-item list on a fifty-item pipeline is a data problem rather than a busy day. `max_active_tis_per_dag` on the mapped task so even a legitimate large day is metered. And a pool for whatever the mapped task talks to, so the blast radius is bounded to that resource rather than to the cluster.

The recovery, since it will be asked: clear or mark the mapped task failed to stop it, fix the upstream, and be aware that forty thousand instances are now in the database — `db clean` afterwards. And if the large list was genuine, the right shape is probably batching: have the upstream return chunks of five hundred, so the fan-out is eighty tasks each doing more work rather than forty thousand doing very little.

The reasoning

The inputs are the peak number of tasks you need running at once — measurable from the task instance history — and the memory footprint of a typical task. Total slots needed is the peak; the split between many small workers and few large ones is decided by the memory profile and by how much blast radius you can tolerate when one worker dies.

`worker_concurrency` is the number of task processes per worker, and it is the setting people leave at the default. Each process needs enough memory for the heaviest task that can land on it, so concurrency times peak task memory must fit the machine — otherwise the OOM killer takes the worker and every task on it, which shows up as a batch of zombies rather than as a memory error.

Then elasticity. A fixed fleet sized for the 9am peak is idle sixteen hours a day; KEDA scaling on Celery queue depth converts that to paying for what you use. The constraint is worker startup time versus how spiky the load is — scale-up that takes two minutes is fine for a daily peak and useless for a thirty-second burst.

The two things that make sizing easier rather than better: push heavy work out to pods or the warehouse so the fleet can be uniformly small, and stagger schedules. A very large fraction of peak load is self-inflicted, because every DAG is scheduled at the top of the hour — spreading them across the hour flattens the peak and is free.

The reasoning

In Airflow 2, every running task opened a connection to the metadata database. That put a hard, unglamorous ceiling on fleet size: a few thousand concurrent tasks meant a few thousand database connections, and the database ran out before the workers did. It also meant every worker needed network access and credentials to your metadata database, which is a security posture nobody liked.

Airflow 3 routes task communication through an API server instead. The database connection count now scales with the API layer rather than with the task count, the credential no longer sits on every worker, and workers can live somewhere the database does not have to be reachable from — which is what makes remote and eventually multi-language execution feasible.

The other scaling-relevant changes: the dag processor is a separate component by default, so a slow parse cannot starve scheduling; and DAG versioning means a run is pinned to the code it started with, which removes a class of mid-run inconsistency that got worse the more frequently you deployed.

What has not changed is the shape of the advice. Pools, `parallelism`, queue routing and deferrable operators work the same way, parsing cost is still the usual scheduler bottleneck, and the metadata database still needs cleaning. So the tuning knowledge transfers; what changes is that one specific ceiling — database connections per task — has moved, which matters if that is what you were hitting and is invisible if it was not.

The reasoning

Policy does not scale and review misses things, so the control has to be structural. A pool per team, sized as that team's share, means the worst a team can do is starve itself — the constraint is enforced by the scheduler, not by anyone remembering. That single change converts a cluster-wide outage into a team-local one.

Then the specific hazard: `max_active_tis_per_dag` on mapped tasks, and a `max_map_length` low enough to fail rather than flood. A CI check that flags a mapped task with no cap is cheap and catches the pattern at review time.

Then priority for the genuinely critical path, and reserved capacity for it if the deadline is a business commitment — priority orders within a pool but does not guarantee, so a small dedicated pool is what actually protects a deadline.

The bigger question worth raising: whether one shared Airflow is the right shape. A single deployment is cheaper and gives you one place to look; separate deployments per team give real isolation of scheduler, workers and database at the cost of duplicated operations. The middle path is one control plane with per-team queues and pools, which is what most multi-team estates settle on — and the deciding factor is usually whether the teams' pipelines actually depend on each other. If they do, keep one deployment and enforce quotas; if they do not, the isolation is nearly free.

The reasoning

Every task return value becomes an `xcom` row that lives until something deletes it. A pipeline returning a few kilobytes per task, over thousands of tasks a day, over two years, is exactly how a metadata database reaches hundreds of gigabytes. Nothing failed along the way — the database simply got slower for everyone, and the scheduler is its heaviest reader.

The immediate remediation is `airflow db clean` with a retention window. That is a large delete on a busy table, so it wants to be done in batches and outside peak hours, and afterwards the table needs the storage reclaimed — a `VACUUM FULL` on Postgres, which takes a lock, so plan it.

The structural fix is to stop putting payloads there: write to object storage and return the key, or configure a custom XCom backend so large values are transparently offloaded and only a URI is stored. Airflow 3 ships object-storage-backed XCom for this. Either way the rule is the same — XCom carries references.

Then make it not recur: a retention policy on `db clean` running on a schedule rather than as a one-off, monitoring on metadata database size so the next growth curve is noticed at 40 GB rather than 400, and a review habit of asking what a task returns. The related tables — `task_instance`, `log`, `dag_run`, `rendered_task_instance_fields` — grow the same way and are cleaned by the same command.

The reasoning

The dag processor parses file by file, with `parsing_processes` files in flight at once. Two thousand small files parallelise well and each one is cheap. Twenty files generating a hundred DAGs each means twenty units of work that cannot be split, and any one of them being slow blocks a whole slot — so the tail latency is much worse.

There is a countervailing cost: two thousand files means two thousand imports, and if they all import the same heavy shared module, that import happens two thousand times per parse cycle. Keeping shared code light, and outside the DAGs folder so it is imported as an installed package rather than re-read, matters more than the file count.

What actually decides it in practice is change frequency. `min_file_process_interval` controls how often a file is re-parsed; DAGs that never change can be parsed rarely. Generated DAGs are usually generated from configuration that changes rarely, so the right move is often to generate the *files* in CI and commit them, rather than generating DAG objects at parse time on every cycle.

And the question underneath: does it need to be two thousand DAGs? Frequently a few hundred DAGs with a task mapped over the varying dimension gives the same coverage with a fraction of the parsing and a much smaller UI. Separate DAGs earn their place when they need separate schedules, owners or failure isolation — not because there are two thousand of something.

The reasoning

Everything scheduled at the same minute produces a thundering herd: every DAG creates a run simultaneously, every task queues at once, the scheduler does all its work in a burst, and the metadata database takes a spike of writes. Then nothing happens for fifty minutes. You are sized for a peak you created.

Spreading them — `7 * * * *`, `13 * * * *`, and so on — flattens the curve with no infrastructure change and no code change beyond the cron string. Where the timing genuinely does not matter, which is most of the time, this is the cheapest capacity win available.

Where the timing does matter, the constraint is usually 'after the source updates' rather than 'on the hour'. That is a dependency, and expressing it as one — a dataset, or a sensor on the actual condition — is better than a cron that guesses, and it naturally spreads the load because sources do not all land at once.

The remaining peak is then real, and worth sizing for properly: autoscaling on queue depth so the fleet grows for the burst, and priority or reserved pools so the pipelines with deadlines are not queued behind the ones without. But do the free thing first — a saturated cluster where every DAG shares one cron minute is a scheduling problem being treated as a capacity problem.

The reasoning

A triggerer runs triggers as asyncio coroutines in a single event loop, so thousands of concurrent waits cost very little as long as each one spends its time awaiting IO. Capacity is configurable — `triggerer.default_capacity` — and triggers are distributed across running triggerers, with a dead one's triggers redistributed. So two is the minimum for redundancy rather than for throughput.

The way to break it is synchronous work inside a trigger. A blocking network call, a CPU-heavy loop, or a library that is not async stalls the whole event loop and every other trigger with it — the symptom is thousands of deferred tasks all becoming unresponsive at once, which looks like the triggerer being down. Writing custom triggers is where this happens; provider ones are generally well behaved.

The other limits worth naming: triggers must be serialisable, since they are persisted and may be picked up by a different triggerer; and the resume path costs a scheduling round trip, so a trigger firing very frequently is not cheaper than a poll. Deferral wins on *waiting*, not on high-frequency polling.

How to know it is the bottleneck: the count of deferred tasks against triggerer capacity, and the delay between a trigger's condition becoming true and the task resuming. If tasks are sitting deferred well past the point their condition fired, either capacity is exhausted or something is blocking the loop, and those are different fixes — more triggerers versus finding the offending trigger.

The reasoning

The one that matters most is *task start delay* — the time between a task becoming eligible and actually running. It is the number users experience, and it degrades before anything fails. Rising start delay with idle workers means a limit; rising with busy workers means capacity; rising with tasks not even reaching `queued` means the scheduler.

Then the scheduler's own health: loop duration and total parse time. These tell you whether the scheduler is keeping up and, since parsing and scheduling compete, which half is consuming it. A parse time creeping up over months is the classic slow degradation, and it is invisible unless it is on a dashboard.

Then saturation: running tasks against `parallelism`, per-pool slot utilisation, and Celery queue depth. Pool utilisation sitting at a hundred percent is the single most actionable signal there is, because it names the resource that is short.

Then the substrate: metadata database size, connection count and query latency, since that is the ceiling large deployments actually hit. And the business-level check that catches everything the technical metrics miss — is each critical output present and fresh by its deadline. If I could keep only two, it would be task start delay and output freshness: one is the leading indicator and the other is the thing that actually matters.