Apache Airflow

What breaks when cron runs a real pipeline, the four guarantees an orchestrator buys you, and the honest cases where Airflow is the wrong tool.

Every component, what it owns, and how one task travels from a DAG file to a finished run — plus what breaks when each piece fails.

DAG vs DAG run vs task vs task instance, the task state machine, and why top-level code in a DAG file costs you every 30 seconds.

A complete runnable DAG, then the exact screens it produces: Grid, Graph, Gantt, logs — and what clearing a task really does.

Why a daily DAG for Monday runs on Tuesday. Data intervals, logical dates, catchup, timezones, and the start_date mistakes everyone makes once.

Core and provider operators, poke vs reschedule sensors, hooks and connection reuse, and how to write a custom operator that is worth writing.

The @task decorator, dependencies inferred from function calls, and an honest account of when classic operators are still the better answer.

Why XCom is metadata and not data, the pass-a-pointer pattern, and every template macro rendered against a concrete logical date.

The connection model, secrets backends, and the single Variable.get() call in top-level code that can bring a scheduler to its knees.

Generating DAGs from config, .expand() and .partial() fan-out, and the point where dynamic generation stops being clever and starts being unmaintainable.

Why the join task after a branch silently never runs, every trigger rule in one place, and TaskGroups for DAGs people can actually read.

Running a backfill without flooding the source system, retry and timeout settings that behave, and the Deadline Alerts that replaced SLAs in Airflow 3.

Retries, backfills and clears all mean your task will run twice. Partition overwrite, MERGE, atomic swaps — and how to prove a task is safe.

How each executor actually runs a task, what it costs, and a decision framework that starts from your workload shape rather than fashion.

The full concurrency ladder, pools that protect your warehouse, and converting idle sensors into deferrable tasks that free their worker slot.

Assets (renamed from Datasets in Airflow 3), producer/consumer DAGs, and the same cross-DAG dependency solved three ways so you can pick on evidence.

Airflow triggers work; it is not a compute engine. Submitting to Spark, EMR and Databricks, KubernetesPodOperator, and dependency isolation.

DAG integrity tests that catch a broken deploy, unit-testing logic outside operators, and how DAG code actually reaches production.

Remote logging, the metrics that actually predict incidents, zombie tasks, and a symptom → diagnosis → fix table you can use on call.

MWAA vs Composer vs Astronomer vs self-hosted, what Airflow 3 changed, and the deprecations that still bite when you follow an old tutorial.