Airflow Debug & Fix

The reasoning

Airflow discovers DAGs by importing every file in the dags folder. If the import raises — a syntax error, a missing provider package, a module that is not on the path — the file produces no DAG. There is nothing to display, nothing to schedule and nothing to mark red. The absence *is* the error, which is why people stare at an empty UI looking for a failure.

`airflow dags list-import-errors` prints exactly the file and the traceback, and it is the first command to run for this symptom. `airflow dags list` shows the same thing indirectly — no data, plus a hint that files failed to load. The dag processor also logs it, and its log is where you would find it in a deployment you cannot get a shell on.

The most common causes in order: a provider package installed in your local environment but not in the image the scheduler runs; a relative import or a helper module that is not on `PYTHONPATH`; a genuine syntax error from a bad merge; and a parse that times out because of slow top-level code, which is the nasty one because the DAG appears and disappears intermittently.

The prevention is the fifteen-line DAG-integrity test in CI: build a `DagBag`, assert `import_errors` is empty. It runs in seconds and it turns this from an hour of confusion into a failed pull request. That the check is so cheap is why the absence of it is worth flagging in an interview.

The reasoning

First the trivial ones, because they are also the most common. Is the toggle actually on — a newly deployed DAG is paused by default unless `dags_are_paused_at_creation` says otherwise. Is `start_date` in the future, or dynamic, in which case no interval ever completes and no run is ever due.

Then the schedule itself. A `@daily` DAG deployed this morning has not yet reached the end of its first interval, so 'no runs today' is correct behaviour and the first run appears tomorrow. A cron with a typo — `0 0 * * 7` — can be valid and mean something you did not intend. And `schedule=None` means it only ever runs when triggered.

Then the limits. If `max_active_runs` is reached and existing runs are stuck in `running`, no new run can start; the DAG looks idle and is actually blocked. `depends_on_past` does the same thing at task level: one failed run silently blocks every subsequent one, and nothing is red because the tasks are simply not eligible.

Then the scheduler. If it is dead, overloaded, or the DAG file's parse is timing out, nothing is scheduled for anyone — so check whether *other* DAGs are producing runs. That single question splits 'my DAG is misconfigured' from 'the cluster is broken' immediately, and it is the check most people skip.

The reasoning

Start with the framing, because it is the answer the interviewer wants: Airflow *will* run your task more than once. Retries, clears, backfills and zombie-task rescheduling are all normal. A task that cannot tolerate that is broken regardless of which of them triggered it.

The specific causes, in likelihood order. A retry after the task wrote rows and then failed — the most common by far. Someone clearing the task to 'try again' during an incident. A backfill for a date range that overlapped a scheduled run, so two runs wrote the same partition concurrently. And a task that was marked as a zombie because its worker died mid-write, then rescheduled while the original work was actually still in flight.

The genuinely-Airflow cause is running two schedulers without the HA support that arrived in 2.0, or with a metadata database that does not support the row locking it relies on. On a modern, correctly configured deployment this is rare, and reaching for it first is usually a way of avoiding the idempotency conversation.

The fix is always the same and it is not 'turn off retries': make the write own its partition. Delete-then-insert for the interval in one transaction, `INSERT OVERWRITE`, a MERGE on the key, or a file path derived from the logical date. Then add a `unique`-style check downstream so a duplicate fails loudly rather than being averaged into a dashboard.

The reasoning

The state is the diagnosis. `scheduled` would mean the scheduler had not released it yet. `queued` means it has — the executor now owns it. That immediately rules out dependencies, trigger rules and the scheduler, which is where most people start looking.

Then work down the capacity chain. Is `parallelism` — the cluster-wide cap on running tasks — already reached? Is the DAG at `max_active_tasks`? Is the task in a pool with no free slots, which is the classic when a hundred sensors in `poke` mode hold every slot? Is `max_active_tis_per_dag` limiting a mapped task?

Then the workers themselves. With Celery: are any workers alive and subscribed to the queue this task was routed to? A task sent to queue `gpu` with no gpu worker running sits queued forever and looks identical to a capacity problem. Is the broker reachable? With Kubernetes: is the pod pending because the cluster cannot schedule it — no nodes, resource requests too large, image pull failing? `kubectl get pods` answers that in seconds.

The two that catch people out. A task queued and then never picked up can be an executor that lost it — `task_queued_timeout` exists to detect that and re-queue. And if the tasks are running but you cannot see it, the worker may be unable to write logs to remote storage, which makes a running task look stuck.

The reasoning

A running task heartbeats into the metadata database. If the process disappears — OOM kill, node termination, a spot instance reclaimed, someone restarting a worker — nobody updates the row, so it stays `running` forever. The scheduler's zombie detection looks for task instances whose heartbeat is older than `scheduler_zombie_task_threshold` and fails them, which lets the normal retry logic take over.

So a zombie is not a bug in itself; it is Airflow recovering from a worker that vanished. The bug is whatever killed the worker. The causes worth naming: memory — a task loading a large DataFrame is the usual suspect, and the OOM killer leaves nothing in the task log, which is why the log looks like it just stops; spot or preemptible instances being reclaimed; a Kubernetes pod evicted; and a worker restart during a deploy.

How to confirm: the task log ends abruptly with no traceback, and the scheduler log has the zombie message. Correlate with worker memory metrics or the node's kill log. A traceback would mean the task failed normally, which is a different problem.

The mitigations. Size workers for the tasks that actually run on them, or better, push heavy work out to a pod or a cluster so a big job cannot kill the worker running twenty small ones. Use `execution_timeout` so a hung task fails cleanly rather than being detected as a zombie much later. And drain workers before deploys. The important consequence is that a zombie means a retry, so this is another route to a task running twice — which is only safe if it is idempotent.

The reasoning

`depends_on_past` makes a task instance wait for the same task in the *previous* dag run to have succeeded. If run N never succeeded — it failed, was cleared and left, or is still in `running` — then N+1 waits, N+2 waits, and so on indefinitely. The runs sit in `running` with tasks in `scheduled` or `none`, which no default alert covers.

Finding it: sort the DAG's runs by logical date and look for the oldest one that is not `success`. That is the blocker. Fix or clear it, or mark the offending task instance successful if the work was genuinely done, and the queue drains in order.

The same shape appears with `wait_for_downstream`, which additionally requires the previous run's downstream tasks to have finished — a bigger surface to get stuck on. And a related version has nothing to do with either: `max_active_runs` reached with runs stuck in `running` blocks new runs identically, and also without anything being red.

The prevention is monitoring the right thing. Alerting only on task failures misses this entirely, because nothing fails. Alert on staleness — 'this DAG has not had a successful run in N intervals' — or on the freshness of the table it produces. That check catches this, a paused DAG, a dead scheduler and a misconfigured schedule, all of which are invisible to failure-based alerting.

The reasoning

Expanding over an empty list produces no task instances, so Airflow has nothing to run and marks the mapped task `skipped`. That is reasonable on its own. What follows is the same skip propagation as after a branch: `publish` has a skipped upstream, `all_success` is not satisfied, so `publish` is skipped too.

The run is `success`, because skips are not failures. So a day when your vendor sent nothing looks exactly like a day when everything worked — and if `publish` was the task that refreshes a dashboard or notifies downstream, that notification silently did not happen.

Whether that is right depends on the pipeline, and being explicit about it is the answer. If an empty day is genuinely normal, you want `publish` on `none_failed_min_one_success` so it still runs — or a guard task that short-circuits deliberately. If an empty day means the feed is broken, you want it to *fail*: a check task that raises when the list is empty converts a silent skip into an alert.

The lesson to generalise: `skipped` is a third outcome that most monitoring ignores. Alerting on failures catches neither this nor a branch that took the wrong path. If the pipeline's job is to produce something, the durable check is on the output — did the table get rows, is it fresh — rather than on the task states.

The reasoning

Connections are looked up by `conn_id` when the task runs, in a defined order: any configured secrets backend first, then `AIRFLOW_CONN_*` environment variables, then the metadata database. So 'works locally' usually means you have an env var or a local database row that production does not, or production's secrets backend has a different value under the same id.

The checks, in order. Does the `conn_id` exist in the environment the worker runs in — `airflow connections get <id>` from a worker, not from your machine. Is the secrets backend configured and reachable; a throttled or misconfigured backend produces a lookup failure that reads like a missing connection. Does the worker's network actually allow the egress — a correct connection to a host the worker cannot reach fails at connect time and looks similar.

Then the subtler ones. Different Airflow images between scheduler and worker, so the provider package supplying the hook is missing on one. A connection whose `extra` differs between environments — an SSL flag, a role, a warehouse name — so the connection succeeds and the query fails on permissions. And credentials that have simply expired or rotated, which is the most common of all and the reason a secrets backend with rotation beats rows in a database.

The prevention: never rely on a connection existing locally. Keep both environments resolving through the same mechanism — a secrets backend with different scopes — so 'works on my machine' stops being a category of difference. And add a lightweight connectivity check to the deployment so a missing or expired connection is found at deploy time rather than at 3am.

The reasoning

Airflow renders only the attributes an operator lists in `template_fields`, and only just before execution. An attribute not in that tuple is whatever Python assigned it — here, a string containing braces. Nothing errors, because a filename with braces in it is a perfectly valid filename.

The two operators beside this are identical except for that one line, and the returned values show it: one wrote to `s3://reports/{{ ds }}/out.csv`, the other to `s3://reports/2026-03-05/out.csv`. Every run of the un-templated one wrote to the same path, so each day silently overwrote the last and there is only ever one file.

The fix is `template_fields = ('target',)` on the operator class. If the value is a path to a file you want loaded and rendered, `template_ext = ('.sql',)` does that too — which is how long queries live in `.sql` files instead of Python strings.

How to catch this class of bug generally: look at rendered templates in the UI, which shows exactly what each field became for a given task instance, before blaming the code. And prefer the TaskFlow style where you can — a `@task` function that takes `data_interval_start` as a parameter gets a real datetime object, and there is no string to forget to render.

The reasoning

Airflow's contract is simple: the task callable raises, or it succeeded. A `try/except Exception: pass` around a database call converts a total outage into a green tick, and the downstream tasks then run happily on nothing. There is no mechanism by which Airflow could know better.

It is almost always well-intentioned — someone added it because a transient error was paging people, and the fix for noise became a fix for signal. The correct response to a flaky call is retries with backoff, which Airflow already provides, not swallowing the exception.

Where catching genuinely is right, the task must still make the outcome visible: re-raise after logging, raise `AirflowSkipException` if the correct outcome is a skip, or raise `AirflowFailException` to fail immediately without burning the retries on something that will never succeed. Those three cover nearly every legitimate case, and all of them keep the state honest.

The systemic answer, because catching this by code review alone does not scale: assert on the outcome rather than on the run. A downstream check task that fails when the table has no rows for the interval, or a freshness check on the target, turns 'green and empty' into a red task. That check would have caught this regardless of what the Python did.

The reasoning

A task that computes its window from `now()` is a function of when it runs, not of what it is for. On the day it was written that is indistinguishable from correct. On a backfill, sixty runs execute within minutes of each other, so all sixty compute the same window and write the same data to sixty different partitions.

The same bug bites on a retry — a task that failed at 23:58 and retried at 00:02 crosses a day boundary and fetches the wrong day — and on any run that is delayed by a queue.

The fix is to derive everything from the context: `data_interval_start` and `data_interval_end` in the TaskFlow signature, or `{{ ds }}` / `{{ data_interval_start }}` in a templated field. A useful rule to state: a task should be a pure function of its logical date, and `now()` appearing anywhere in a DAG file deserves a comment justifying it.

How you would find it after the fact: the partitions all contain the same date range, or the row counts are identical across days that should differ. And how you would catch it before: grep for `now()`, `today()`, `CURRENT_DATE` and `utcnow` in the repo, which takes a minute and is worth doing once on any codebase you inherit.

The reasoning

XCom values are rows in the metadata database, serialized. The value column is sized by backend, and exceeding it fails the task at push time, after all the work is done — which makes it especially annoying, because the expensive part succeeded.

The right fix nine times out of ten is to stop passing data. Write the DataFrame to S3 under a key derived from the dag id, task id and logical date, and return the key. The downstream task reads it. That is cheap, the metadata database stays small, and the intermediate result is inspectable afterwards, which is a debugging bonus.

The mechanism for when you genuinely want it to feel transparent is a custom XCom backend: subclass `BaseXCom`, serialize large values to object storage and store only the URI. Airflow 3 ships object-storage-backed XCom for exactly this. Everything continues to look like `transform(extract())` while the payload never touches the database.

What not to do: raise the column size. It works and it degrades the database every component shares — the scheduler is querying that table constantly, and a table full of megabyte blobs makes everyone's Airflow slower for reasons nobody will connect back to this decision.

The reasoning

Airflow keeps all timestamps in UTC internally and applies your DAG's timezone when computing the schedule. If you set `pendulum.timezone('Europe/London')` and schedule at 02:30, the clock in that zone genuinely repeats 02:00–03:00 in autumn and skips it in spring. A scheduled time that does not exist cannot fire; one that occurs twice may fire twice.

The fix that removes the class of problem is to schedule in UTC and convert for presentation. Business logic that needs local dates can still compute them, but the *schedule* should be on a monotonic clock. If the run genuinely must happen at a local business hour, pick a time well away from the transition window — 04:30 rather than 02:30 — so the ambiguity cannot bite.

The related bug is more common and quieter: a task filtering on a local date against data stored in UTC. That is off by an hour for half the year and produces boundary rows landing in the wrong day — the same signature as the month-end mismatch that timezone bugs always produce.

Two Airflow specifics worth knowing. `start_date` must be timezone-aware; a naive datetime is interpreted against the configured default and will surprise you. And the UI has historically displayed times in UTC while people read them as local, which is its own source of confusion when reconciling a run against a log from another system.

The reasoning

A `poke`-mode sensor is a running task for its entire wait. Twenty DAGs each starting with a sensor at 9am is twenty occupied slots, and on a cluster with twenty-four you have four left for everything else. Nothing has failed; the capacity is simply spoken for. The self-resolution is the tell — the sensors succeed as their files land, and the backlog drains.

Confirming it: look at what is actually in `running` during the stall. If it is mostly sensors, you have your answer in ten seconds. The other candidates for a cluster-wide queue — `parallelism` set too low, a pool sized smaller than intended, workers down — do not have the daily rhythm or the spontaneous recovery.

The fixes in increasing order of goodness. `mode='reschedule'` frees the slot between pokes and is a one-line change. A dedicated pool for sensors caps how much of the cluster they can ever hold, which protects everything else even if someone adds twenty more. Deferrable sensors move the wait into the triggerer entirely, so thousands of waits cost one process — that is the real answer at scale, and it needs a triggerer deployed.

The design question underneath: why is anything waiting six hours for another team's file? A dataset trigger, an event from the producer, or the producing pipeline calling yours removes the wait rather than making it cheaper. Sensors are for genuinely external systems, and even then they should have a `timeout` so a file that never arrives fails rather than waiting forever.

The reasoning

The scheduler alternates between parsing DAG files and making scheduling decisions. If parsing the whole folder takes minutes, the interval between scheduling passes stretches, and everything is uniformly late. That uniformity is the diagnostic: one late DAG is a dependency or a pool, *everything* late is the scheduler.

Confirming it: the scheduler exposes the parse duration per file and the total loop time — `airflow dags report` lists the parse time of every file, and the biggest numbers are your answer. The metrics to watch in a real deployment are `dag_processing.total_parse_time` and `scheduler.scheduler_loop_duration`.

The usual causes, in order. Top-level code doing real work — an API call, a `Variable.get`, a database query, a heavy import — repeated on every parse of every file. Too many DAG files, or a few enormous ones. A metadata database that has become slow, often because nobody has ever run `airflow db clean` and the task instance and log tables have millions of rows.

The fixes: move work inside tasks, use templated `{{ var.value.x }}` instead of parse-time lookups, raise `min_file_process_interval` if the DAGs genuinely change rarely, run more scheduler replicas (supported since 2.0), and clean the metadata database on a schedule. The last one is the most commonly neglected and often the biggest single win on an older deployment.

The reasoning

The default is that the worker writes to its own filesystem and the webserver reads from there. In Kubernetes or on autoscaled workers, that filesystem is gone as soon as the pod or instance is — so the log exists only while the worker does. This is why remote logging to S3, GCS or Elasticsearch is effectively mandatory on any containerised deployment, and its absence is a deployment bug rather than an Airflow one.

If remote logging is configured and logs are still missing, check the worker's ability to write there: credentials, bucket permissions, and the `remote_base_log_folder` path. A worker that cannot write logs runs tasks perfectly well and silently produces nothing to read, so the tasks look fine until one fails.

If the log exists but stops abruptly with no traceback, the process was killed rather than having failed — OOM, an evicted pod, a preempted instance. That is the zombie signature, and the evidence is in the node or container runtime rather than in Airflow. Correlate with the scheduler log, which will have recorded the zombie detection.

Where else to look while you are blocked: the scheduler log for the state transitions, the executor's own log for whether the task was ever dispatched, and — with Kubernetes — `kubectl logs` on the pod if it still exists, or the cluster's event log if it does not. And for the future, ship task logs to the same place as everything else so a task failure can be correlated with what the target system was doing at that moment.

The reasoning

The first mechanism is concurrency. A backfill creates many runs at once; if `max_active_runs` allows several and the task writes to a shared target — a whole table rather than a partition — two runs interleave and the last writer wins. Today's scheduled run and a backfill run for 3 March can be writing at the same moment.

The second is a window bug. If the task's filter is not derived from the interval — the `now()` mistake — every backfill run writes today's data, which means the backfill actively overwrote correct partitions with wrong ones. That is the worst version, because the damage is proportional to how much history you backfilled.

The third is a task that truncates. `TRUNCATE` followed by an insert of the interval's rows is idempotent for a single run and catastrophic under concurrency: run A truncates while run B has just inserted. It looks fine in testing and only fails when two runs overlap.

The prevention is the same discipline as always, applied with concurrency in mind: each run owns exactly one partition and touches nothing else; `max_active_runs=1` for anything writing a shared target; and backfill with `--max-active-runs` set explicitly rather than relying on the DAG's value. Afterwards, reconcile — compare row counts per partition against a source of truth before assuming the backfill worked.

The reasoning

Start with the rendered template for a recent task instance. That tells you the literal query that ran, including the date window. A window that is empty, in the wrong timezone, or literally containing `{{ ds }}` explains it immediately, and all three are common.

Then check the source for that window directly, outside Airflow. If the source has no rows either, the pipeline is honest and the problem is upstream — which is what source freshness monitoring is for, and its absence is the real finding.

Then check the destination. A task writing to a different schema or a different environment's table than the one you are looking at is embarrassing and frequent, especially where connections differ between environments. And a write inside a transaction that is never committed succeeds from Python's point of view and persists nothing.

Then the code path. An exception caught and ignored produces exactly this. So does a task that iterates a paginated API and silently gets an empty first page, or a filter that is always false because a parameter defaulted to something no row matches.

The systemic fix is to stop treating task success as evidence. Add an assertion to the pipeline — rows written for this interval is greater than zero, or the target's max timestamp advanced — so 'succeeded and did nothing' becomes a failed task. That check is a few lines and it converts a week of silence into an alert on the first night.

The reasoning

First, check the task instance's `max_tries` rather than the DAG source. If it is zero, the configuration never reached the task: `retries` set in `default_args` after the tasks were created, an operator that sets its own, or the DAG being edited after this run's task instances were created — task instances capture their configuration when created, so an old run does not pick up a new setting.

Second, the exception type. Raising `AirflowFailException` fails the task immediately and deliberately skips remaining retries — it exists for errors that will certainly not succeed on a second attempt, and someone may have used it correctly. `AirflowSkipException` similarly ends in `skipped` rather than failing.

Third, the failure may not have come from the task at all. A task killed by `execution_timeout`, or one whose run was cleared or marked failed by a human, ends up in `failed` without the retry path being taken in the way you expect. And a `dagrun_timeout` failing the whole run stops everything regardless of per-task retries.

Fourth, and easiest to miss: it may have retried and you are looking at the last attempt. The try number on the instance and the log dropdown show the earlier attempts; three fast failures with `retry_delay=timedelta(seconds=0)` look like one failure in the UI unless you check.

The reasoning

Scope first, because it changes everything else. Are other DAGs running normally? If nothing is running anywhere, this is infrastructure — scheduler, database, workers — and the finance pipeline is a symptom. If everything else is fine, it is this pipeline.

Then find the *earliest* failure in the run, not the one someone noticed. A screen full of red is usually one `failed` task and a cascade of `upstream_failed` ones; only the first has a useful log. Sort by start time and read that task's log — and read it to the end, because the actual exception is often above a wall of retry noise.

Then classify before acting. Transient — a timeout, a 503, a lock — means clear the task and watch. Data — a bad row, a schema change, a missing file — means the retry will fail identically and you need the upstream owner. Code — something merged yesterday — means revert rather than retry. And 'not failed at all', where the DAG is idle because of `depends_on_past`, a paused toggle, or a full `max_active_runs`, which is why the scope check comes first.

Then act with the blast radius in mind: clear the failed task with downstream if the fix is upstream of them, note the logical dates affected so a backfill can follow, and leave a comment on the incident with what you saw — because the person who picks it up at 8am needs the log line you have already found. What I would not do at 3am is mark tasks successful to make the UI green; that discards the record of what actually needs reprocessing.