Airflow Operational Scenarios

The reasoning

Before anything: confirm idempotence for a past date. Seven hundred runs of a task that appends is seven hundred duplicated partitions, and you will not notice until someone queries it. Run one day, diff the output against what is already there, then proceed.

Then bound the concurrency. `airflow backfill create --dag-id x --from-date --to-date` with an explicit max active runs, or a dedicated pool with a small number of slots that only the backfill uses. Without one, the backfill will happily consume every worker and the nightly run queues behind two years of history.

Then chunk it. A month at a time gives you checkpoints, lets you verify output before continuing, and means a mistake costs a month rather than two years. It also keeps the metadata database from taking seven hundred dag runs at once, which on a busy deployment is its own problem.

Two refinements worth offering. If the DAG has `depends_on_past`, the backfill is forced serial and will take a long time — consider whether that flag is genuinely needed for historical reprocessing, and if it is, plan for the duration. And if the work is warehouse-shaped, the fastest backfill is often not Airflow at all: one SQL statement over the whole range beats seven hundred orchestrated runs, and Airflow's job is then just to record that it happened.

The reasoning

Each run owns exactly one schedule interval, so the granularity of a backfill is the granularity of the schedule. For an hourly DAG, `--from-date 2026-03-01 --to-date 2026-03-02` is twenty-four runs, and each task instance sees a one-hour window in its `data_interval_start`/`data_interval_end`.

That matters for two practical reasons. First, cost and time: twenty-four runs of a task with a two-minute startup is nearly an hour of overhead before any work happens, which is why chunking an hourly DAG's backfill by month is a very different proposition from a daily one's. Second, correctness: if the task's write is keyed on `ds` rather than on the full interval, all twenty-four runs of a day write to the same partition and the last one wins — a bug that only appears on hourly schedules.

The boundary question that follows: `--to-date` semantics. Getting a range off by one interval leaves a gap or reprocesses an extra window, and on an hourly DAG the gap is an hour of missing data that nothing will alert on. Verify by listing the runs the backfill created and checking the first and last logical dates against what you intended, before assuming it covered the range.

The general habit: after any backfill, reconcile rather than trust. Count rows per interval in the target against the source for the same window. That is the check that catches an off-by-one range, a non-idempotent write and a wrong-window bug all at once.

The reasoning

Clearing is the normal action: it removes the task instance's state, the scheduler sees it as eligible again, and it executes. You can clear downstream too, which is what you want when the fix changes the output that downstream tasks consumed. That is the button for 'the cause is fixed, do it properly'.

Marking success writes `success` without running anything. It is right in exactly one situation: the work is genuinely done — you ran the load by hand, the vendor re-sent and you loaded it manually — and you need the graph to reflect reality so downstream can proceed. It is wrong whenever the work has not happened, because it converts a visible failure into an invisible gap.

The failure mode of over-using it: the run goes green, the pipeline continues, and a partition is empty. Nothing will ever tell you, because the state says it succeeded. That is why 'mark success to make the UI green' is a habit worth arguing against explicitly at 3am.

Two related actions. Clearing with downstream re-runs the subtree, which is what you want after fixing an upstream bug; clearing without it leaves downstream tasks holding old results. And clearing a task in a *past* run re-runs it for that logical date — which is only safe if the task is idempotent, so the same discipline underpins the whole operational surface.

The reasoning

Catchup is a property of the DAG, evaluated by the scheduler. With it on, the gap between `start_date` and now is filled automatically — on first deploy, and again any time the DAG is paused and resumed. That second case is the one that surprises people: pausing a DAG for a week and unpausing it produces a week of runs.

A backfill is an operator action with a bounded range and its own concurrency controls. You choose the dates, you choose how many run at once, you can stop it. That is the property you want for anything expensive, which is why `catchup=False` plus explicit backfills is the standard production posture.

The exception where catchup is right: a DAG that genuinely must process every interval and where missing one is a data gap — an incremental load with no other recovery path. Even then, `max_active_runs=1` so it serialises, and a start date that does not claim more history than you intend.

The operational note that follows: with catchup off, a pause is a data gap. If you pause a DAG for maintenance, you are choosing to skip those intervals, and somebody has to backfill them afterwards. Making that explicit — a note in the runbook, or an alert on missing partitions — is what stops a two-hour maintenance window becoming two hours of permanently missing data.

The reasoning

This is the whole point of connections. A DAG says `conn_id='warehouse'`, and the credential is resolved at task runtime from the secrets backend, an environment variable, or the metadata database — in that order. Forty DAGs referencing one `conn_id` means one thing to change.

With a proper secrets backend — Secrets Manager, Vault, GCP Secret Manager — the rotation happens in that system and Airflow picks up the new value on the next lookup. You do not restart anything, you do not deploy anything, and the rotation is audited where rotations belong. If there is caching configured, the cache TTL is the only lag.

Without one, the connection is a row in the metadata database and someone updates it through the UI or `airflow connections`. That works, and it is worse: no audit, no rotation policy, a manual step, and the credential now exists in a database backup. Environment variables via `AIRFLOW_CONN_*` are the cheap middle ground — no extra infrastructure, and rotation means redeploying the workers.

The failure to plan for: tasks running *at* the moment of rotation hold the old credential and will fail. On a rotation you either accept a few retries — which is fine if retries are configured, and is a good argument for having them — or you schedule it in a quiet window. And check for anything that reads the credential at parse time or caches a client at module level, because that will keep using the old value until the process restarts.

The reasoning

The two models. Baking DAGs into the container image makes the deployment atomic and versioned — every component runs exactly the code in that image — at the cost of a full deploy for a one-line DAG change. Syncing from git onto a shared volume makes changes fast and decouples DAG authors from platform deploys, at the cost of a propagation window and a shared filesystem to operate.

The window is the real hazard. With git-sync, the scheduler may pick up a new DAG file seconds before a worker does, so a task can be scheduled against a task id that the worker's copy does not have — which produces confusing failures that resolve themselves a minute later. The same applies to any change that alters the task set of a DAG mid-run.

What makes it safe: never edit a DAG in a way that changes its task set while a run of it is active, if you can avoid it; keep deployments atomic where possible; and use Airflow 3's DAG versioning, which records the version each run used, so 'which code did this run execute' stops being guesswork.

The rest of the pipeline matters as much as the mechanism. CI should run the DAG-integrity test so a file that does not parse never reaches the scheduler, since a broken DAG deploys silently and disappears rather than failing. Dependencies belong in the image rather than the DAGs folder, because a DAG importing a package the workers do not have is the same class of failure with a longer feedback loop.

The reasoning

Airflow renders a dag run against the current DAG definition, so adding a task makes it appear in historical runs with no state — the grid shows gaps going back three years. Nothing breaks; the runs are complete as far as their own state is concerned. But it looks like three years of missing work, and someone will eventually clear one of them and find out whether the new task is safe to run for a date from 2023.

For the run in progress, the scheduler picks up the new definition on its next parse. If the run has not finished, the new task can be scheduled within it — with dependencies as written — which means a run that started under one definition finishes under another. Where the new task is downstream of things that already succeeded, that generally works; where the edit changes existing dependencies, it can produce a state nobody designed.

Removing a task is the mirror image and slightly worse: historical instances still exist in the database but the task is gone from the DAG, so the UI shows them oddly and clearing that run cannot recreate it. Renaming a task id is a remove plus an add, which is why renames lose history.

The safe practice: make task-set changes when no run is active, prefer adding over renaming, and if you must rename, accept that history is broken and say so. Airflow 3's DAG versioning improves this materially, because a run is pinned to the version it started with rather than being re-rendered against whatever is current.

The reasoning

The timetable derives the interval from the cron, so changing the cron changes what each run covers. Under `0 6 * * *`, the run labelled 1 March covers 06:00 on the 1st to 06:00 on the 2nd. If your partitions are keyed on `ds`, the partition named `2026-03-01` now holds six hours of the 1st and eighteen of the 2nd — same name, different data.

The transition run is the awkward one. The last midnight-based run and the first 06:00-based run either overlap or leave a gap, depending on which direction you moved. That gap is real missing data and nothing will alert on it, so the change needs a deliberate backfill or an accepted, documented gap.

What to check before merging: does anything downstream join on the partition label and assume it means a calendar day; does the source system's own daily boundary still line up; and does the change cross a business-relevant cutoff, such as a financial close.

The alternative that avoids the whole problem: keep the schedule and the partitioning independent. Derive the partition from a business timestamp in the data rather than from the run's label, and the run can move whenever operations needs it to. That decoupling is worth arguing for, because schedules move for operational reasons and data boundaries should not follow them.

The reasoning

The hour-later schedule is a guess that fails silently. On a day your run takes seventy minutes, their DAG reads yesterday's table and reports success. Nothing is red, and the wrongness is proportional to how much they trust the number.

With datasets, your producing task declares `outlets=[Dataset('warehouse://analytics/fct_orders')]` and their DAG uses `schedule=[Dataset(...)]`. When your task succeeds, Airflow triggers theirs. You never reference their DAG; they never reference yours. A third consumer can subscribe with no change on your side, which is what makes this scale across teams.

The alternatives and why they are worse here. `TriggerDagRunOperator` in your DAG couples you to them and means editing your pipeline whenever their team changes. `ExternalTaskSensor` in theirs blocks a worker slot and requires the schedules to align exactly, which reintroduces the timing coupling you were removing.

The caveat to be honest about: a dataset is marked updated when the task *succeeds*, not because Airflow inspected the data. A task that succeeds having written nothing still triggers them. So the producing task should assert its own output — rows written for this interval greater than zero — or the contract is only as good as the task's honesty. That is a good thing to agree explicitly when the consumer is another team.

The reasoning

The position is the answer. A check task placed between staging and publishing means a bad file stops the pipeline: the check fails, `publish` becomes `upstream_failed`, and the previous good data remains in place. A check that runs after publishing is a report, not a gate.

What to check, in rough order of value: volume against a recent baseline — today's rows within a sensible band of the trailing average catches truncated files, which is the case here; freshness of the source; not-null and uniqueness on the key; and referential checks against a dimension. `SQLColumnCheckOperator` and `SQLTableCheckOperator` from the common-sql provider cover most of this without custom code, and dbt tests do if the transformation is in dbt.

The trade-off to name: a strict gate turns a data problem into an outage, and an alert-only check turns it into noise nobody acts on. Which you want depends on whether stale-but-correct beats fresh-but-wrong for the consumer — for finance it usually does, for an operational dashboard it may not. Saying that explicitly is the senior part of the answer.

The pattern that gives you both: publish to a staging location, check, then swap. The check gates the swap rather than the whole pipeline, so a failure leaves the previous version live and gives you the bad data to look at. That is more machinery than most pipelines need, and it is the right answer for the ones where being wrong is expensive.

The reasoning

The rule I use: page when a human must act *now*, alert to a channel when a human must act today, and log everything else. A task failure that has two retries left is not a page — it may well fix itself before anyone opens a laptop.

What earns a page: an output the business depends on will not exist by its deadline. That is best expressed as a freshness check on the target — 'the finance table has no data for yesterday and it is 07:00' — rather than as a DAG failure, because it catches every cause at once: the failed task, the DAG that never ran, the paused toggle, the dead scheduler, and the task that succeeded having done nothing. Failure-based alerting misses four of those five.

What goes to a channel instead: final task failures, SLA misses, data quality warnings, and backfill completions. What is logged and dashboarded only: retries, individual queue depth, parse times — these are for capacity work, not for interrupting someone.

The part people forget: alert on the *absence* of runs. Every failure-based alerting scheme is blind to a DAG that stopped being scheduled, and that is one of the most common real outages — a paused DAG, a blocked `depends_on_past` chain, a full `max_active_runs`. A single 'no successful run in N intervals' check covers all of them and is usually the highest-value alert in the estate.

The reasoning

Every component reads and writes it. The scheduler decides what to run from it, workers report state to it, the UI renders from it, and it holds connections and variables. It is the one stateful thing in an Airflow deployment and it deserves the treatment any production database gets: backups, monitoring, connection pooling, and a plan for restoring it.

Losing it means losing history and state. Your DAG code is safe — it is in git — so you can rebuild an Airflow that runs, but every completed run, every XCom, and every connection stored there is gone. Practically, that means the next scheduled run happens normally and you have no record of what did or did not process. The mitigation for the connections half is a secrets backend, so credentials do not live there in the first place.

The maintenance nobody does is `airflow db clean`. Task instance, log, XCom and dag run tables grow forever, and on a busy deployment they reach tens of millions of rows within a year or two. The symptom is gradual: the UI gets slow, the scheduler loop stretches, and nobody connects it to the database because nothing failed. Running `db clean` on a retention policy is one of the highest-value pieces of Airflow maintenance and it is almost always missing.

The other pressures worth naming: connection count, because every scheduler, worker and triggerer holds connections and a large Celery fleet can exhaust the database's limit; and write volume from heartbeats, which is why very short heartbeat intervals are expensive at scale. If the database is the bottleneck, those two are usually why.

The reasoning

Sequence the risk. Get to the latest 2.x release with current providers first — most breakage is provider changes rather than core, and doing them separately means you are never debugging two things at once. Run the upgrade check tooling to get the machine-detectable list.

Then the code sweep for what was removed: `execution_date` in code and templates, `SubDagOperator`, direct imports of `airflow.models` inside tasks (which no longer work because task execution goes through an API), pickled XComs, and anything calling the Airflow 2 REST API. The database-access one is the biggest rewrite and the easiest to miss, because it is a pattern rather than a name.

Then the scheduling decision, made explicitly. Airflow 3 builds a `CronTriggerTimetable` from a cron string by default, so any DAG whose SQL uses `data_interval_start`/`data_interval_end` gets a zero-width window and silently returns nothing. Either keep data-interval behaviour for those DAGs or rewrite their windows — but decide per DAG rather than discovering it in production.

Then run them in parallel. Stand up 3.x against a copy, deploy the DAGs with catchup off, and compare outputs for a few days on the DAGs that matter before cutting over. Migrate in waves by team or criticality so a problem affects a subset. And plan the metadata database migration itself — it is a one-way schema change, so the rollback plan is a restore, which means a tested backup before you start.

The reasoning

What you genuinely get: someone else runs the scheduler, the metadata database, the workers and the webserver; upgrades are a supported path rather than a project; and there is a support contract when it breaks. For a small data team with no platform engineers, that is a large fraction of the operational burden gone.

What you do not get: your DAGs are still yours to design, test and debug. Dependency management is often *harder* — MWAA installs from a `requirements.txt` with constraints and a failed install can be opaque; Composer has its own conventions. Version availability lags, so you may wait months for a release you want. And cost is frequently higher and less elastic than an equivalent self-hosted deployment, particularly with an idle environment running all night.

The constraints that bite in practice: limited control over Airflow configuration, restricted network setups, log access through the provider's tooling rather than yours, and the executor choice being made for you. If you need a custom executor or an unusual deployment shape, managed will fight you.

How I would decide: if there is no platform team and Airflow is not core to what you differentiate on, managed almost always wins — the alternative is a data engineer spending a day a week on Kubernetes. If you already run Kubernetes well, have opinions about executors, or have cost or compliance constraints, self-hosting on the official Helm chart is very reasonable. And there is a middle path worth mentioning: self-host the control plane and push all heavy work to external compute, so the Airflow deployment itself stays small and boring.

The reasoning

First, an inventory of reality rather than of intent. Which DAGs have had a successful run recently, which are paused, and which have not produced a run in weeks. That last group is where the silent outages are — a `depends_on_past` chain blocked in March, a DAG someone paused for maintenance and never resumed. `airflow dags list` plus a query over dag runs gives you this in an hour and it is always more interesting than people expect.

Second, whether anyone would know if it broke. Where do alerts go, does that channel have a human reading it, and is there any check on outputs rather than on task states. A deployment with no output-level alerting is one where the failures you inherit have already happened and nobody noticed.

Third, the operational hygiene: has `db clean` ever run, how large are the task instance and log tables, is remote logging configured (or do logs vanish with the workers), where do credentials live, and are DAGs deployed from a repository with any CI at all. The DAG-integrity test is usually the first thing I would add, because it is fifteen lines and it stops the most common silent failure.

Fourth, the risk list rather than the tidiness list. Which pipelines feed something the business would notice within an hour; whether those are idempotent, since that decides whether you can safely fix them under pressure; and what the recovery story is if the metadata database is lost. I would deliberately not spend the first week refactoring DAGs — the goal is to know what exists, what is broken, and what would hurt, and only then to start changing things.

The reasoning

Pausing is reversible and is the right first step: scheduling stops immediately, history is intact, and if it turns out somebody depended on the output you can unpause. Leave it paused long enough for a quiet consumer to complain — a month is not unreasonable for anything with a monthly cycle.

Removing the file stops it being parsed, so it vanishes from the DAG list, but its dag runs, task instances and XComs remain in the metadata database. Depending on version and configuration it may linger in the UI as an inactive entry. So 'delete the file' is not 'delete the DAG'.

`airflow dags delete <dag_id>` removes the metadata — runs, task instances, XComs. It is destructive and unrecoverable, which is why it comes last and only after you are sure nobody needs the history. On a pipeline whose runs are an audit record, you may want to export before deleting, or simply never delete.

The order, then: pause, announce, wait, remove the file through a pull request so the removal is reviewed and reversible in git, then delete the metadata once you are confident. And check for dependents before starting — anything with an `ExternalTaskSensor` on it, any dataset subscribers, and any downstream table that quietly stops updating. A grep of the DAGs folder for the dag id takes a minute and prevents the most embarrassing version of this.

The reasoning

The convenience is real: a Variable can be changed in the UI or via the API without a deploy, which is what the requirement asks for. The hazard is that a DAG file is re-imported every parse interval, so `Variable.get('regions')` at module scope becomes a database query every thirty seconds per DAG — four DAGs is a constant load on the database everything else depends on.

Reading it at runtime removes the problem entirely. Inside a `@task`, `Variable.get` runs once per task instance. In a templated field, `{{ var.value.regions }}` is rendered just before execution. Either is correct; neither touches the database during parsing.

The case where this gets hard is when the config decides the *shape* of the DAG — one task per region. That genuinely needs the value at parse time, and the answer there is not a Variable but a file in the repository, so a region change is a reviewed pull request. Or better, keep the DAG shape fixed and map a task over the regions at runtime, which moves the decision back to execution time where it belongs.

The judgment to state: config that changes weekly and is operationally significant deserves review. A Variable is a change with no diff, no reviewer and no history — convenient during an incident and a poor place for something four pipelines depend on. Where I would use a Variable is a toggle or a threshold; where I would use the repository is anything that changes what gets built.

The reasoning

Start with impact, because it decides whether the person should act at all: what does this produce, who consumes it, what is the deadline, and what happens if it is late versus wrong. 'Finance close, needed by 07:00, stale is better than incorrect' tells a half-asleep engineer more than three paragraphs of architecture.

Then the recovery, as commands rather than as prose. Which task to clear and whether to include downstream. Whether a rerun is safe — that single line, backed by the task actually being idempotent, is the difference between confident action and paralysis. What to do if it fails again identically.

Then the known failures with their signatures: 'if the log ends with no traceback, the worker was OOM-killed, rerun and it usually passes'; 'if the vendor file is missing, do not rerun, it will not appear before 06:00, escalate to X'. These are what make a runbook worth more than the UI, and they can only come from having had the incidents.

Then the boundaries: when to escalate and to whom, with a name and a channel; what you must not do — marking tasks successful to clear the board, running a backfill without checking concurrency; and where the dashboards and logs are. What I would leave out is architecture diagrams and anything that will rot. A runbook that is out of date is worse than none, so it should be short enough that keeping it current is trivial, and updated by whoever handles the incident it did not cover.

The reasoning

The scheduler, webserver and database are a fixed, modest cost. Where money goes is workers — usually provisioned for the daily peak and idle the rest of the time — and whatever those workers do. So the first split to make is: what does the Airflow control plane cost, and what does the work it orchestrates cost. They are usually an order of magnitude apart, and people optimise the wrong one.

The specific patterns that inflate it. Sensors in `poke` mode occupying workers for hours doing nothing, which is paying full price for waiting — deferrable operators move that to one cheap triggerer process. Heavy compute in the worker, forcing every worker to be sized for the largest task; pushing that to a pod or a warehouse means the fleet can be small. And a backfill nobody bounded, which quietly doubled the month.

Then the elasticity question. A fixed worker fleet sized for the 9am peak is idle for sixteen hours a day. KEDA-based autoscaling on Celery queue depth, or the Kubernetes executor's scale-to-zero, converts that into paying for what you use — at the cost of pod startup latency, which is the trade to weigh against how short your tasks are.

How to find it rather than guess: tag the compute Airflow launches with the dag id and task id so the cloud bill can be grouped by pipeline, and use `run_results`-style duration data from the metadata database to rank tasks by total runtime. In every deployment I have seen, a handful of tasks account for most of the cost, and the conversation becomes about those rather than about Airflow.

The reasoning

The two options and their consequences. Pause the DAG and no runs are created for that window — with `catchup=False` those intervals are simply never processed, which is a data gap you must backfill deliberately. Leave it running and the runs happen, fail, retry, and either recover on their own or exhaust their retries inside the outage and stay failed.

If the data matters, the cleanest plan is: pause before the window, let maintenance happen, unpause after, then backfill the missed range explicitly. That gives you a bounded, verifiable reprocessing rather than a pile of failed runs to clear by hand.

If you leave it running, tune the retries so they outlast the outage — three retries with a forty-five-minute exponential backoff spans two hours, where three retries at five minutes burns out in fifteen and leaves everything red. That is a legitimate approach for a pipeline that is genuinely resilient, and it means no manual step afterwards.

Either way, tell the alerting. A window of expected failures that pages the on-call is how people learn to ignore the pager. And afterwards, verify rather than assume: check that every interval in the window has data, because the failure mode of this plan is a couple of runs quietly missing while everything looks recovered.

The reasoning

**What Airflow is actually for.** Dependencies that span systems — this Spark job after that API extract after that dbt run, with a sensor waiting on a vendor file. Its second real advantage is **backfill as a first-class operation**: a parameterised logical date, and the ability to re-run an arbitrary historical window. Managed workflow services orchestrate their own vendor's tasks well and are noticeably weaker at both.

**What it costs.** A scheduler, workers, a metadata database, an upgrade path, and someone who knows why a task is stuck in `queued`. That is a real operational burden even on MWAA or Composer, where you still own DAG performance, pools, concurrency and the database growing without bound. If nobody owns it, it degrades quietly.

**Step Functions** is the right answer for event-driven workflows with modest branching, especially where the steps are already Lambdas and the state machine is short. It is serverless, priced per state transition, and it has no scheduler to operate. It gets expensive and unreadable when the workflow has hundreds of steps or a dynamic fan-out.

**Glue Workflows** is right when everything is already Glue: it orchestrates Glue jobs and crawlers with no extra infrastructure. It is weak the moment a dependency leaves that boundary, which for most teams is soon.

**And the third option people forget:** the warehouse's own scheduler, or dbt Cloud, when the entire pipeline is transformations inside one warehouse. dbt already knows the dependency graph — running Airflow on top of it to trigger one command per model is orchestration theatre, and one `dbt build` in a scheduled task is the honest version.

**The decision, stated simply:** how many systems does one workflow touch, do you need historical re-runs, and is there someone to operate a scheduler. Two yeses and Airflow pays for itself. One and it is worth arguing about.

The reasoning

**First, check whether it is one incident.** Two failures five minutes apart usually share a cause — a schema change upstream, an expired credential, a region degradation, a shared cluster out of capacity. Establishing that costs two minutes and can turn two problems into one. Working them as separate incidents when they share a root cause means fixing the symptom twice.

**Then rank, and say the criteria out loud.** *Blast radius* — how many consumers, and are any of them customer-facing. *Reversibility* — a pipeline that is silently writing wrong data outranks one that has cleanly stopped, because the stopped one is recoverable and the running one is getting worse every minute. *Deadline* — regulatory or contractual cut-offs are hard, an internal dashboard is not. *Cost of delay* — some failures are equally expensive to fix at 3 a.m. and at 9 a.m., and those go second.

**Reversibility is the criterion people miss**, and it frequently inverts the obvious ranking. A failed job that halted is stable: nothing is degrading while you work the other one. A job that is running and producing bad output is actively contaminating downstream tables and every consumer reading them. **The first move on that one is not to fix it, it is to stop it** — halt the write, and that is often enough to demote it while you work the other.

**Communicate both immediately, before fixing either.** Two lines: what is broken, what the impact is, what you are working first and why. That does the ranking work for everyone else — the stakeholder of the second incident stops paging you, and if their case is genuinely more urgent than you judged, they will say so in the next minute rather than in the post-mortem.

**Ask for help early rather than late.** One person working two P1s serially is slower than two people working one each, and the cost of waking a colleague is much lower than the cost of the second incident sitting untouched for an hour. Escalating is a decision, not an admission.