Executors Compared: Local, Celery, Kubernetes

The last chapter said an executor sends a task out to run. One line in the config picks which kind of executor you use.

That one line also decides what you must deploy, and what you pay for. This chapter compares the main kinds, then shows how to choose.

You follow the same run as before: shop_orders_daily, the run about 2026-01-20, dag_run 380. It has two tasks, load_orders and then build_daily_revenue.

You also meet a made-up company with 40 DAGs. Most of them start at midnight. Every number about that company is made up.

This chapter is about Airflow 2. Notes about Airflow 3 are marked. You answer three questions:

1. Which executors exist, and what does each one need?
2. How long does a task wait under each one?
3. How do you choose, and how big must it be?

The simplest executors run tasks on the scheduler machine. A machine is one computer or one server.

SequentialExecutor runs one task at a time. In Airflow 2 it is the only executor that works with SQLite, the test database from the last chapter. Use it only for learning.

Airflow 3.0 removed SequentialExecutor. There, LocalExecutor also works with SQLite for testing.

LocalExecutor starts each task as a subprocess. A subprocess is a small program that another program starts. Here the scheduler starts it on its own machine.

Several subprocesses run at the same time. The setting parallelism is the limit: how many tasks can run at once. Its default is 32.

Look at the config. Then look at where the two tasks of our run ran:

-- airflow.cfg
-- [core]
-- executor = LocalExecutor
-- parallelism = 32

SELECT task_id, state, hostname
FROM   task_instance
WHERE  dag_id = 'shop_orders_daily'
  AND  run_id = 'scheduled__2026-01-20T00:00:00+00:00';

That is the strength of LocalExecutor: nothing else to run. It is also the limit. You cannot add a second machine.

CeleryExecutor sends tasks to workers on many machines. Celery is a Python tool for running background jobs.

It needs a broker, a message queue such as Redis or RabbitMQ. It also needs a result backend, a place where workers save results. In many setups this is the metadata database.

Celery workers are long-running. A worker starts once, then waits for tasks and runs many of them. So a task starts fast: our run waited about 2 seconds.

Each worker runs several tasks at once. The setting worker_concurrency says how many. Its default is 16.

You can also split the fleet with queues. A queue is a named line of tasks.

Each worker listens to the queues you give it. Start two workers like this:

# on machine worker-a
airflow celery worker --queues default

# on machine worker-b
airflow celery worker --queues heavy

Now send a task to a queue. Say build_daily_revenue needs more memory, so it goes to heavy:

load = PythonOperator(task_id="load_orders", python_callable=load_orders)
build = PythonOperator(
    task_id="build_daily_revenue",
    python_callable=build_daily_revenue,
    queue="heavy",
)
load >> build

KubernetesExecutor starts one new pod for each task. A pod is a small, isolated box that Kubernetes runs. A cluster is the group of machines that runs the pods.

The pod is built from an image: a saved package with your code and libraries. A file called pod_template_file describes the pod. It names the image, the memory and the CPU.

When the task ends, Airflow deletes the pod by default. With no tasks running, no task pods exist.

So no idle workers wait for tasks. You still pay for cluster machines unless the cluster shrinks.

Look at a small template. The image name is made up:

apiVersion: v1
kind: Pod
metadata:
  name: placeholder-name
spec:
  restartPolicy: Never
  containers:
    - name: base
      image: my-registry/shop-airflow:1.4
      resources:
        requests:
          cpu: "500m"
          memory: "1Gi"

Every task gets this pod unless you change it. You change it for one task with executor_config. Here build_daily_revenue asks for more memory:

from kubernetes.client import models as k8s

build = PythonOperator(
    task_id="build_daily_revenue",
    python_callable=build_daily_revenue,
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(requests={"memory": "4Gi"}),
                    )
                ]
            )
        )
    },
)

Isolation also covers libraries. Each task can use a different image. Two tasks that need different versions of one library can live in the same DAG.

This is not the same as KubernetesPodOperator. There, the work of one task runs in a container you choose. Running External Compute: Spark, Docker & Kubernetes Pods teaches it.

Now run the same two tasks under each executor. A task waits between queued and running. That wait is the cost of the executor.

Each task works for 3 seconds. The waits in the table are made up. Only the order is typical: a process is fastest, a waiting worker is close, and a new pod is slowest.

Check the Kubernetes row. load_orders is queued at 00:00:02 and runs at 00:00:22. It ends at 00:00:25, after 3 seconds of work.

build_daily_revenue is queued at 00:00:27. It runs at 00:00:47 and ends at 00:00:50. The run ends at 00:00:51.

Only 6 of those 51 seconds are your task code. For one daily run, nobody cares. For thousands of short tasks, the waits add up.

The wait also changes where you look when a task is stuck in queued. Look at the places:

Start from your work, not from a fashion. Four facts decide most cases:

• How many tasks run at once.
• How long each task runs.
• Whether libraries conflict.
• What your team can run and fix.

A library conflict means two tasks need different versions of the same library. One shared environment cannot hold both. Look at how the three executors differ:

Short tasks favor Celery, because workers are already running. Long or very different tasks favor Kubernetes, because the start-up wait is small next to the work.

Try one case. A made-up task runs for 3 seconds and waits 20 seconds for its pod. Only 3 of 23 seconds are work, so Celery wins.

Try the other case. A task runs for 40 minutes and needs its own library version. A 20-second wait does not matter, and the isolation does, so Kubernetes wins.

Measure your real task lengths before you decide.

Now the made-up company. It has 40 DAGs with 5 tasks each, so 200 tasks. Each task takes 2 minutes.

Most tasks become ready at midnight. To keep it simple, pretend no task waits for another, so all 200 are ready together. A wave is one group of tasks that run together.

With Celery, capacity is workers times worker_concurrency. Say 4 workers have 16 slots each, so 64 slots. Do 64 tasks run at once?

No. The core setting parallelism is also a limit, and its default is 32.

A pool is another limit: a set of task slots. Look at four setups:

Check row 1. 200 divided by 32 is 6.25, so 7 waves. Seven waves of 2 minutes is 14 minutes, and half of the 64 slots stay empty.

Row 3 shows the trap. The extra 4 workers changed nothing, because parallelism was still 64. In row 4, 13 workers give 208 slots, because 200 divided by 16 is 12.5 and you round up.

The last row costs 13 workers all day for a peak that lasts 2 minutes. KubernetesExecutor changes this deal. It could start 200 pods at midnight, if the cluster has room, and then use none.

Parallelism and the pool still apply, so they must allow 200 too.

Add the made-up 20-second pod wait. The peak then takes about 2 minutes 20 seconds, and no task pods run after it.

This needs worker_pods_creation_batch_size raised, because its default of 1 starts one pod per scheduler loop.

Other limits also cap a DAG or a shared system. Concurrency, Pools, Priorities & Deferrable Operators teaches them.

Most teams have a main kind of task and a few odd ones. Say 195 small SQL tasks and 5 heavy jobs that need a big pod.

Airflow 2.10 added multiple executors. You list several in the config, and the first one is the default.

A task picks another one with its executor argument.

Look at the config and at our DAG. Here build_daily_revenue is the odd task:

# airflow.cfg  (Airflow 2.10 or later)
# [core]
# executor = CeleryExecutor,KubernetesExecutor

load = PythonOperator(task_id="load_orders", python_callable=load_orders)
build = PythonOperator(
    task_id="build_daily_revenue",
    python_callable=build_daily_revenue,
    executor="KubernetesExecutor",
)
load >> build

Before 2.10, the only way to mix was a hybrid executor: CeleryKubernetesExecutor or LocalKubernetesExecutor. They sent tasks with one queue name to Kubernetes. From Airflow 2.10, a list of executors does the same job.

Airflow 3.0 does not support them. If you use one, move to a list of executors before you upgrade.

An executor is one line of config and a different deployment. Choose it from your tasks, then check the limits above it.

Look at the answers you should know:

Try this alone. Your tasks run 10 seconds each, thousands of times a day. Which executor fits, and which number decides it?

The answer is Celery. The start-up wait decides: a made-up pod wait of 20 seconds is longer than the task.