APACHE AIRFLOWScale & Production

Running External Compute

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Airflow holds a job id and a status. Everything with real memory requirements happens somewhere it can be sized independently.

⏱ 4 min · Topic 1 of 6

Chapter 2 called Airflow a control plane and said data should not flow through it. This chapter is that rule applied to compute: the heavy work belongs on Spark, Databricks, EMR or a container, and Airflow’s job is to submit it and wait for a verdict.

Set the dataset size below and watch the point where “just do it in the worker” stops being viable.

Your task processes this much data. Where should it actually run?
Dataset size40 GB
In the Airflow worker40 GB will not fit in an 8 GB worker. The task is OOM-killed, or you size every worker for the largest task and pay for that memory all day.
On external computeAirflow submits the job to Spark, Databricks, EMR or a pod, waits for the verdict, and records the result. The worker holds a few megabytes of Python for the whole thing.
This belongs on external computeChapter 2 called Airflow a control plane. This is where that rule earns its keep: the moment you size workers for data volume rather than for coordination, you are paying for a compute cluster that also happens to schedule things — and doing both badly.
Core mental model

Airflow holds a job id and a status. Everything with real memory requirements happens somewhere it can be sized independently.

Why it matters

The moment you size workers for data volume rather than for coordination, you are running a compute cluster that also schedules things — and doing both worse than two dedicated systems would. It is also the most expensive shape, because those large workers sit idle whenever the heavy task is not running.

Control plane
The layer that decides what runs and when. Handles metadata; not data.
Submit-and-poll
The pattern for remote work: hand the job over, then wait for its outcome.
KubernetesPodOperator
Runs a task as a container in a cluster. Works under any executor, not just KubernetesExecutor.
Common mistake

Loading a large dataset into a worker because pandas is convenient. Either the task is OOM-killed, or you size every worker for the largest task and pay for that memory around the clock. Both are worse than submitting the work to something built for it.

Better habit

Ask of every task whether it is coordinating or computing. Computing usually belongs elsewhere.

Size workers for coordination overhead. Needing large workers is a signal, not a requirement.

Keep small transforms in the worker — the rule is about scale, not about purity.

Trigger the work, do not be the work

Chapter 2 in one line. Almost every “Airflow is slow” or “Airflow runs out of memory” report resolves to compute sitting in the wrong layer.

Remember this

Airflow submits and waits. The compute belongs somewhere you can size for it.