What You’ll Master Here
The executor decides how a task travels from “ready” to “running”. Everything downstream of that — brokers, workers, pods — is real infrastructure you own.
The authoring track is behind you. From here the question changes from “what should this pipeline do?” to “what does it take to run thousands of them reliably?” — and the first decision shapes everything else.
The executor is one line of configuration and an entirely different deployment. Compare the four below, paying attention to the “you must deploy” row: that is the real cost.
- Runs the task: A message goes to a broker; long-lived workers pull from it.
- You must deploy: A broker (Redis or RabbitMQ) plus a worker fleet you keep running.
- Start latency: Seconds
- Isolation: Per worker — all tasks on a worker share its image
- Scaling: Add workers. Well understood, and the most common production choice.
- Reach for it when: Many short-to-medium tasks, a stable workload, and a team happy to operate a broker.
The executor decides how a task travels from “ready” to “running”. Everything downstream of that — brokers, workers, pods — is real infrastructure you own.
Chapter 2 established that the executor is an interface inside the scheduler rather than a service you run. What it implies, though, is infrastructure: a broker and a worker fleet, or a Kubernetes cluster and pod templates. Choosing it is choosing what your team operates for the next few years.
- Local executor
- Runs tasks inside the scheduler process. Fast and simple; task load and scheduling share a machine.
- Remote executor
- Decouples execution from the scheduler. Adds scalability, and adds latency and components.
- EdgeExecutor
- A remote executor whose workers pull tasks over HTTP from outside the cluster — for work that must happen elsewhere.
Choosing an executor because a conference talk recommended it. Kubernetes for five hundred two-second tasks spends most of its time pulling images; Celery for a bursty nightly load means paying for idle workers all day. The workload decides, not the fashion.
Start from the workload shape — task duration, burstiness, dependency conflicts — and let the executor follow.
Remember that CeleryKubernetesExecutor and LocalKubernetesExecutor were deprecated in Airflow 3.0.
Treat an executor change as an infrastructure project, because that is what it is.
executor = CeleryExecutor is a line of config. It is also a broker, a worker fleet, autoscaling, and a new set of things that can break at 3am.
The executor is the deployment decision disguised as a config value. Choose from the workload, not from what others run.