APACHE AIRFLOWScale & Production

Managed Airflow, Airflow 3 & Migration

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Your DAGs are portable across all four options. Your deployment tooling, networking and upgrade cadence are not.

⏱ 4 min · Topic 1 of 7

Two decisions close the Scale and Production track, and both sit above everything else in it. Should your team be operating Airflow at all? And if you are on Airflow 2, how do you get to 3 without breaking your pipelines?

Start with the first. Compare what each option actually obliges you to run — the “you operate” row is the honest comparison.

Four ways to run Airflow
Self-hosted
  • You operate: Everything — scheduler, database, workers, upgrades, backups.
  • Upgrades: Whenever you choose. You control the version and the timing entirely.
  • Cost shape: Infrastructure plus real engineering time.
  • Lock-in: None. It is the upstream project.
  • Choose it when: You already run Kubernetes well and want full control over version and configuration.
Your DAGs are portable across all four — they are just Python. What is not portable is deployment tooling, networking and any provider-specific integration you lean on.
Core mental model

Your DAGs are portable across all four options. Your deployment tooling, networking and upgrade cadence are not.

Why it matters

Self-hosting looks cheaper on every price list and frequently is not, because the comparison usually omits the upgrade weekends, the database tuning, the on-call rota and the person who becomes the Airflow expert by accident. Counting those changes the answer for a lot of teams.

MWAA
Amazon Managed Workflows for Apache Airflow. AWS-native, with the upgrade cadence AWS chooses.
Cloud Composer
Google’s managed Airflow, running on GKE with a management fee on top of the cluster.
Astronomer
A managed Airflow vendor employing many Airflow committers; typically the fastest to new versions.
Common mistake

Comparing managed pricing against infrastructure cost alone. You compare a service bill against a server bill and conclude self-hosting wins, having priced the engineering time at zero. It is usually the largest term.

Better habit

Count engineer-days honestly, including the ones spent on upgrades and incidents.

Check the version cadence before committing — waiting a year for a release is a real constraint.

Keep your DAGs free of vendor-specific tooling so the decision stays reversible.

The DAGs are portable; the operations are not

Moving between managed services is mostly a deployment-tooling exercise. That is the argument for keeping configuration in connections and variables rather than in vendor-specific mechanisms.

Remember this

The comparison is not price against price. It is price plus engineering time against price.