Airflow Conceptual

The reasoning

Airflow schedules intervals, not instants. A `@daily` DAG divides time into one-day windows, and the run for the window starting 1 March cannot start until that window has closed — otherwise it would be summarising a day that had not finished. So `logical_date` is 2026-03-01 and the scheduler queues it at 2026-03-02 00:00.

That is why the name changed. It used to be `execution_date`, which everyone read as 'the moment this executed', and it is not — it is the label of the period the run is responsible for. Airflow renamed it to `logical_date` precisely because the old name taught the wrong thing, and both names still appear in older code and blog posts.

The practical consequence is the one that matters in an interview. Your task must derive its window from the context — `data_interval_start`/`data_interval_end`, or `ds` — and never from `datetime.now()`. A task that queries 'yesterday' relative to wall-clock time produces the right answer on the day it runs and the wrong answer on every backfill and every retry, and nothing in Airflow will tell you.

The table beside this is Airflow's own timetable enumerating the runs it would create. Note the last column: the gap between logical_date and the moment the scheduler queues it is exactly one schedule interval, which is why a weekly DAG appears to be a week behind and a monthly one a month.

The reasoning

The interval is the schedule period, whatever that period is. For `0 6 * * 1` it is one week ending at the next Monday 06:00, so the run labelled 2 March covers 2 March 06:00 through 9 March 06:00 and is queued at 9 March 06:00. Seven days elapse between the label and the execution, which is what makes weekly DAGs feel broken to anyone holding the daily intuition.

`data_interval_start` and `data_interval_end` are the two values your query should actually use — `where event_at >= '{{ data_interval_start }}' and event_at < '{{ data_interval_end }}'` is the correct, backfillable filter, and the half-open range is deliberate so adjacent runs neither overlap nor leave a gap.

`ds` is a convenience: the logical date as `YYYY-MM-DD`, which equals `data_interval_start` truncated to a day. It is fine for a daily DAG and actively misleading for a weekly or hourly one, because it throws away the part of the interval that matters. The habit worth having is to reach for the interval values and treat `ds` as a partition label rather than as a filter.

The gotcha to name before you are asked: on the *first* run of a DAG there is nothing before the start date, and on a manually triggered run the interval is inferred rather than scheduled. So code that assumes a full previous interval exists will behave differently for manual triggers — which is exactly the difference between the timetable table above and what a `dags test` run produces.

The reasoning

The scheduler's job is to make sure a run exists for every completed interval between `start_date` and now. With `catchup=True` it does exactly that on unpause: sixty dag runs appear at once, and they begin executing up to `max_active_runs` at a time — which defaults to 16 in Airflow 2 and is easy to leave unset.

That is fine when the DAG is genuinely backfillable and you meant it. It is a problem when the DAG sends emails, calls a rate-limited vendor API, or rebuilds a table that sixty concurrent runs will fight over. The failure is usually not Airflow falling over — it is the downstream system.

The controls, in order of bluntness. `catchup=False` means only the most recent interval runs, and you backfill deliberately when you want history. `max_active_runs=1` keeps catchup but serialises it, which is the right setting for anything that writes to a shared target. A `start_date` that is genuinely recent avoids the question entirely, and there is no reason for a new DAG to claim it starts a year ago unless you want that history.

The recommendation worth stating: `catchup=False` as the default for new DAGs, plus an explicit `airflow backfill` when you need history. It makes the expensive operation something a human asks for rather than something a deploy does.

The reasoning

Airflow re-parses your DAG file constantly — every thirty seconds by default. `datetime.now() - timedelta(days=1)` is evaluated at parse time, so each parse produces a slightly later start date. The scheduler asks 'has an interval completed since the start date' and the answer keeps being no, because the start date moved. The DAG sits there looking fine and never produces a run.

The fix is a static, timezone-aware literal: `pendulum.datetime(2026, 3, 1, tz='UTC')`. A start date should be a fact about when this pipeline's data begins, not a computed value, and it should never change once runs exist — moving it later leaves orphaned runs, moving it earlier invites a catchup you did not ask for.

A future start_date is less dramatic and worth knowing: the DAG is valid, appears in the UI, and simply produces no runs until the first interval after that date completes. People report it as 'my DAG isn't triggering' and the answer is that it is behaving exactly as configured.

Two adjacent traps in the same family. A naive `datetime` with no timezone is interpreted against the configured default and will surprise you across a DST boundary — always pass a tz. And in Airflow 2 a per-task `start_date` was allowed and would silently override the DAG's for that task; it was a reliable source of confusion and is best treated as something you never set.

The reasoning

The loop, in order. Parse DAG files — in Airflow 3 that is a separate dag processor — and write the serialized form to the metadata database. Create dag runs for any interval that has completed and does not have one. For each running dag run, examine every task instance and decide whether its dependencies are met: upstream states satisfy its trigger rule, its pool has a free slot, concurrency limits allow it, retry delay has elapsed. Those that pass are set to `scheduled` and then `queued`.

The handoff matters. Once a task is `queued`, the scheduler has done its part — the executor is responsible for getting it onto a worker. That is precisely why 'stuck in queued' is almost never a scheduler problem: the scheduler already decided the task should run and something on the execution side has not picked it up.

The scheduler also does the housekeeping that catches people out: detecting zombie tasks whose worker died without reporting, timing out tasks that exceed `execution_timeout`, firing SLA misses, and running the callbacks attached to dag runs.

What it deliberately does not do is execute your code. Nothing in your task body runs in the scheduler — with one important exception that is the source of most performance problems: top-level code in a DAG file runs on every parse, in the parsing process. An API call at module level is a call every thirty seconds, forever.

The reasoning

LocalExecutor runs tasks as subprocesses on the same machine as the scheduler. It is genuinely fine for small deployments — parallelism is bounded by that one host, there is no broker to operate, and a lot of teams over-engineer past it too early. It has no isolation and no horizontal scale.

CeleryExecutor puts tasks on a broker — Redis or RabbitMQ — and a fleet of persistent workers consumes them. You get horizontal scale and queue-based routing, so heavy tasks can go to big workers and light ones to small. The costs are the broker, the worker fleet as a thing to operate, and the fact that workers are long-lived, so a dependency conflict between two DAGs is a real problem.

KubernetesExecutor launches a pod per task instance. That gives perfect isolation — per-task images, per-task resources — and scale-to-zero when idle. It costs pod startup latency on every task, which is brutal for thousands of short tasks, and it needs a cluster and someone who understands it. The hybrid most large shops land on is Celery for the many small tasks plus KubernetesPodOperator for the few heavy or dependency-conflicting ones.

The thing to say that shows operational experience: choose on task shape, not on scale slogans. Many short tasks favour long-lived workers; few heavy or isolation-hungry tasks favour pods. And in Airflow 3 the distinction softens, because task execution goes through a task execution API rather than direct database access — which is what finally makes remote and multi-language execution reasonable.

The reasoning

A DAG file is Python that gets imported repeatedly to discover the DAG object. In Airflow 2 that is the scheduler's parsing loop; in Airflow 3 it is the dedicated dag processor. Either way, everything at module scope executes on every parse: imports, API calls, `Variable.get`, database queries, file reads.

So a `Variable.get('config')` at the top of a DAG file is a metadata-database query every thirty seconds per DAG. Fifty DAGs doing it is a hundred queries a minute against the database that everything else depends on, and the symptom is scheduler lag that nobody can attribute.

The fixes are all about deferring the work to execution time. Move the call inside the task body. Use a Jinja template — `"{{ var.value.config }}"` is rendered when the task runs, not when the file is parsed. Keep heavy imports inside functions. If a DAG genuinely must be built from external configuration, cache it or generate the DAG files in CI rather than fetching at parse time.

Two numbers worth knowing for the follow-up: `min_file_process_interval` controls how often a file is re-parsed, and `dagbag_import_timeout` kills a parse that takes too long — which is how a slow top-level API call turns into DAGs disappearing from the UI intermittently.

The reasoning

`PythonOperator` is a template for work. `load_orders = PythonOperator(task_id='load_orders', ...)` inside a DAG is a task — a node in the graph, with an id unique in that DAG. `load_orders` for the run whose logical date is 5 March is a task instance, and that is the row that has a state, a try number, logs and a duration.

Everything operational is instance-level. Clearing a task clears instances, and it is the act of deleting their state so the scheduler reschedules them. Retries increment a counter on the instance. A mapped task expands into many instances distinguished by `map_index`, all sharing one task id.

This is why 'the task failed' is ambiguous and interviewers listen for precision: the task is fine, one of its instances failed. The same task can be green for yesterday and red for today, and a backfill produces hundreds of instances of it.

The follow-up this sets up: clearing a task instance is not the same as marking it success. Clearing makes Airflow run it again — with its downstream, if you ask for that. Marking success fabricates a state without doing the work, which is right when you fixed the data by hand and wrong almost every other time.

The reasoning

When a task returns a value, Airflow serializes it and writes it to the `xcom` table; a downstream task that takes it as an argument reads it back. That is the whole mechanism, and the fact that it is a row in the metadata database is what dictates the rule.

The rule is that XCom carries references, not payloads. An S3 key, a row count, a partition name, a job id — yes. A DataFrame, a list of a million ids, a JSON blob of the query results — no. The metadata database is the component every other part of Airflow depends on, and filling it with data makes the scheduler slow for everyone.

There are hard limits, which is the concrete half of the answer: the value column is a BLOB sized by backend, historically 64 KB on MySQL and up to a gigabyte on Postgres, and exceeding it fails the task at push time. A custom XCom backend — writing to S3 or GCS and storing only the URI — is the sanctioned way to pass something larger, and it is worth naming because it shows you know the limit is configurable rather than absolute.

Two details that come up. XComs are scoped to a dag run by default, so pulling from a different run means asking explicitly and is usually a design smell. And in Airflow 3 the object-storage XCom backend and the task execution API make large-value handling cleaner, but the design rule — pass a pointer — has not changed.

The reasoning

Every operator declares `template_fields` — a tuple of attribute names. Before running a task instance, Airflow renders those attributes as Jinja against the run's context. `SQLExecuteQueryOperator` templates `sql`, `BashOperator` templates `bash_command` and `env`. Anything not in that list is used as-is.

So `sql="select * from t where ds = '{{ ds }}'"` works, and `my_var = "{{ ds }}"` followed by passing `my_var` somewhere untemplated gives you the literal braces. The failure is quiet: the query runs, matches nothing, and the task succeeds with zero rows.

In the TaskFlow API you generally do not need templating at all — a `@task` function receives the context as keyword arguments, so you take `data_interval_start` as a parameter and use it as a real datetime. That is cleaner than string interpolation and it is type-checked by your editor. Templating matters most when you are configuring a classic operator.

Two things worth adding. You can extend templating to your own operator by declaring `template_fields` on it, and `template_ext` lets a field be a path to a `.sql` file that is loaded and rendered — which is how you keep long queries out of Python. And `{{ params.x }}`, `{{ var.value.x }}` and `{{ conn.my_conn.host }}` are all available in the same render, which is the right way to read a Variable without hitting the database at parse time.

The reasoning

A Connection is a named bundle of host, schema, login, password and extras that hooks resolve by `conn_id`; a Variable is a key-value pair. Both are stored in the metadata database, and connection passwords plus variable values are encrypted at rest with the `fernet_key` — which is why losing that key means losing every stored credential.

The parse-time problem: a DAG file is imported every `min_file_process_interval`, so anything at module scope repeats forever. `Variable.get('config')` there is a query per parse per DAG, which at fifty DAGs is a constant load on the database the scheduler is competing for. The fix is to read it inside the task, or to use the Jinja form `{{ var.value.config }}`, which is rendered at execution time.

In production, neither belongs in the database at all. A secrets backend — AWS Secrets Manager, GCP Secret Manager, Vault, or environment variables — is configured once and Airflow resolves connections and variables through it, with the metadata database as a fallback. That gets rotation, audit and access control from a system built for it, and means a database dump does not contain your credentials.

Two operational details worth having: the environment-variable backend uses the `AIRFLOW_CONN_<ID>` and `AIRFLOW_VAR_<KEY>` naming and needs no infrastructure, which makes it the cheapest real improvement over storing secrets in the UI; and secrets backends are checked before the database, so a backend lookup that is slow or throttled becomes a task-startup latency problem — worth caching.

The reasoning

Every operational action in Airflow assumes it. Retries rerun a task after a partial failure. Clearing reruns it deliberately. A backfill runs it for hundreds of past intervals. If any of those can corrupt the result, you have a pipeline nobody can safely touch during an incident — which is the worst possible time to discover it.

The two failure patterns to name. First, appending: `INSERT INTO fact SELECT ...` duplicates the interval's rows on every rerun. The fix is to make the write replace the partition the run owns — delete-then-insert bounded by the interval, `INSERT OVERWRITE`, a MERGE on a key, or writing to a path derived from `ds` so a rerun overwrites the same object. Second, wall-clock time: a task that filters on `now() - 1 day` computes a different window every time it runs, so a rerun today produces yesterday's answer for a run labelled last March.

Beyond writes, the same discipline applies to side effects. A task that sends an email or posts to Slack is not idempotent by nature, so it belongs behind a guard or at the very end of a DAG, and never in the middle of something that retries.

The test worth naming: run the task twice for the same logical date and diff the target. If the two results differ, it is not idempotent, and no amount of care during the incident will save you. That check is cheap enough to put in CI for the tasks that matter.

The reasoning

The cron guess is the version everyone starts with and it is fragile: it encodes an assumption about A's runtime in B's schedule, so a slow day silently makes B read stale data. Nothing fails. That is the worst property a dependency can have.

`TriggerDagRunOperator` at the end of A is explicit and works, but it couples the producer to its consumers — A now has to know about B, and adding a third consumer means editing A. `ExternalTaskSensor` is the mirror image and worse in a different way: it blocks a worker slot while waiting and needs the two schedules to line up exactly.

Datasets (renamed Assets in Airflow 3) invert the dependency. A task declares `outlets=[Dataset('s3://bucket/orders')]`, and B is scheduled with `schedule=[Dataset('s3://bucket/orders')]`. When A's task succeeds, Airflow marks the dataset updated and triggers B. Neither DAG references the other; you can add a fourth consumer without touching the producer, and the lineage shows up in the UI.

The limits worth stating so it does not sound like a cure-all. A dataset update is signalled by task success, not by Airflow inspecting the data, so a task that succeeds without writing anything still triggers downstream. A DAG scheduled on multiple datasets waits for all of them, and conditional expressions on top of that are newer and worth checking support for. And a dataset-scheduled DAG has no cron interval, which changes what `data_interval` means for it.

The reasoning

It serialises a task across runs. Useful when a task genuinely builds on the previous interval — a running total, a slowly changing dimension, an incremental load whose watermark must advance in order. Without it, catchup can run March and April concurrently and produce nonsense.

The risk is that the dependency is unbounded. If the run for 3 March fails and nobody notices, 4 March waits, 5 March waits, and a week later you have seven blocked runs and a DAG that looks idle rather than broken. It does not alert, because nothing failed — the tasks are simply not eligible.

`wait_for_downstream` is the stronger version: the task waits for the previous run's copy of itself *and everything downstream of it* to succeed. Same risk, larger blast radius, and worth knowing the distinction because interviewers ask for it.

How to use it safely: pair it with `max_active_runs=1` so the ordering is real rather than incidental, alert on dag runs that have been in `running` state longer than expected rather than only on failures, and remember that the very first run has no predecessor — which with catchup off and a manual trigger produces the confusing case where a task will not start and there is no failed run to point at.

The reasoning

`execution_timeout` is enforcement: the task is terminated, it goes to `failed`, and normal retry rules apply. It is what stops a hung connection holding a worker slot indefinitely, and it should be set on anything that talks to a network. The number should be well above the normal runtime — a timeout tuned too tight turns a slow day into an outage.

An SLA is a notification. It is measured from the dag run's start, not from the task's, and when it is missed Airflow records an SLA miss and calls `sla_miss_callback`. The task keeps running. It answers 'is this pipeline meeting the promise we made to the business', which is a different question from 'is this task stuck'.

So the two do different jobs and a mature DAG has both: a generous `execution_timeout` to prevent a wedge, and an SLA reflecting when the downstream consumer actually needs the data. Confusing them gives you either a pipeline that kills itself on a slow day or one that hangs for three hours while nobody is told.

The caveats interviewers like. SLA misses are computed by the scheduler and have historically been unreliable at scale, and the mechanism was reworked in Airflow 3 — so many teams do not rely on it and instead alert externally on 'has the target table been updated by 9am', which is the check the business actually cares about. Saying that shows judgment rather than recital. There is also `dagrun_timeout` for the whole run, which is what you want when the concern is the pipeline as a whole rather than one task.

The reasoning

Compare the two tables. Under data-interval semantics the run labelled 1 March covers 1–2 March and executes on the 2nd; under trigger semantics the run labelled 1 March executes on the 1st, and `data_interval_start` and `data_interval_end` are the same instant. The 'logical date is yesterday' surprise simply does not happen.

That is a better default for the majority of DAGs, which are 'do a thing at 6am' rather than 'summarise the window that just closed'. It is a worse fit for the ones that genuinely are interval-shaped, and for those you keep the old behaviour — the `create_cron_data_intervals` setting controls which timetable a bare cron string produces, or you attach the timetable explicitly.

The migration risk is the part to volunteer. Any task whose SQL filters on `data_interval_start`/`data_interval_end` produces an empty window under trigger semantics, because start equals end. Nothing errors — the query returns zero rows and the task succeeds. That is exactly the kind of silent failure that makes an upgrade dangerous, and grepping for those macros is the first thing to do before migrating.

The rest of the Airflow 3 changes worth naming: tasks execute through a task execution API rather than reaching into the metadata database, which is what enables remote and non-Python execution and hardens the security boundary; the dag processor is a separate component by default; `execution_date` and the old REST API are gone; and DAG versioning means the UI can show you the code a past run actually used.

The reasoning

The one that breaks the most code is that tasks no longer talk to the metadata database directly — they go through a task execution API. Any task that imported `airflow.models` and queried `TaskInstance` or `DagRun` to inspect state, which is a very common hack, stops working. That is deliberate: it is what makes remote execution and non-Python tasks possible and closes a real security hole, but it is a rewrite for the DAGs that relied on it.

Then the straightforward removals. `execution_date` is gone in favour of `logical_date`, so grep for it in code and templates. SubDAGs are gone; TaskGroups replaced them years ago and anything still using SubDAGs needs rewriting. Pickled XComs are gone in favour of JSON-serialisable values or a custom backend. The Airflow 2 REST API is replaced, so anything automating Airflow needs updating.

Then the operational shape. The dag processor runs as its own component rather than inside the scheduler, which is better isolation and one more process to deploy and monitor. The UI is rewritten. DAG versioning means a run records the code version it used, which finally makes 'what did this run actually execute' answerable.

How I would plan it: pin and upgrade providers first, because most of the breakage is actually provider changes rather than core; run the upgrade check tooling; grep for `execution_date`, `SubDagOperator`, direct model imports and the data-interval macros; stand up a parallel Airflow 3 environment and run the DAGs against it with catchup off before switching. And treat the scheduling default explicitly rather than discovering it.

The reasoning

The integrity test is the highest value for the least effort and every project should have it: load a `DagBag`, assert `import_errors` is empty, and assert basic conventions — every DAG has an owner and retries, no duplicate dag ids, tags present. It runs in seconds and it catches the failure that actually hurts, which is a DAG that does not parse and therefore silently vanishes from the UI rather than failing loudly.

Unit tests should target the logic, not the orchestration. Keep task bodies thin and put the real work in plain functions that take arguments and return values; test those with pytest like any other Python. A task whose body is twenty lines of business logic mixed with Airflow context lookups is untestable, and the fix is a refactor rather than a cleverer test.

`airflow dags test <dag_id> <logical_date>` executes a whole run synchronously in-process, which is the closest thing to an end-to-end test. It is genuinely useful against fixture data or a sandbox connection. The caveat to state is that it creates a *manual* run, so its data interval is inferred rather than scheduled — anything that depends on interval semantics behaves slightly differently from production.

What CI cannot easily give you is confidence about connections, permissions and real data volumes. That is what a staging Airflow with production-shaped credentials is for. And a small deploy-time smoke test — trigger the DAG for one date, assert it goes green — catches the class of problem that only appears with real infrastructure.

The reasoning

Airflow decides run state once no task can make further progress. If any task is in `failed` or `upstream_failed`, the run is `failed`; if all finished tasks are `success` or `skipped`, it is `success`. Skips do not make a run fail — a branch skipping half the graph is a normal successful run, which is important because otherwise every branching DAG would look broken.

The `all_done` cleanup case is worth being precise about, because it is the pattern people use for teardown: the cleanup task runs regardless of upstream failure and can succeed, but the failed upstream task is still failed, so the run is failed. That is the behaviour you want — the cleanup happening does not mean the pipeline worked.

The inverse — a green run containing a failed task — is what people actually mean when they ask this, and it happens when a task's failure is absorbed: `trigger_rule='all_done'` downstream plus the failing task itself being marked success by hand, or a task whose failure is caught inside the Python and never raised. A task that swallows its exception and returns normally is `success` no matter what happened, which is the most common way a pipeline lies to you.

Practically: alert on task failures rather than only on run failures, and never write `except Exception: pass` in a task. If a step is genuinely optional, express that with a trigger rule so it is visible in the graph, not with a swallowed exception that is invisible everywhere.

The reasoning

Be honest about what Airflow costs: a scheduler, a database, workers, a deployment pipeline for DAG code, and somebody who understands all of it at 3am. That is a real ongoing burden, and a pipeline that is one linear chain of five steps does not repay it.

What it buys, and when that matters: dependency management across heterogeneous systems, retries with backoff per task, backfilling a date range without hand-writing loops, visibility into where a multi-step pipeline is, and a single place where all scheduling lives. The moment you have twenty pipelines touching six systems with real dependencies between them, the alternative is a pile of cron jobs nobody can reason about.

The alternatives worth naming and what distinguishes them. Cron is fine until you need dependencies, retries or visibility. Dagster centres on data assets rather than tasks and has a stronger local development and typing story. Prefect is lighter to adopt for Python-first teams. Cloud-native options — Step Functions, Cloud Composer, MWAA — trade control for operational burden. dbt Cloud schedules dbt and only dbt, which for a dbt-only shop is exactly enough.

The answer I would give: for this pipeline, trigger dbt from the sync tool or dbt Cloud and skip the orchestrator. Revisit when you have a second pipeline that depends on the first, or a step that is not dbt — that is the point where the graph starts to matter and hand-rolled scheduling starts to fail quietly.