APACHE AIRFLOWScale & Production

Monitoring, Alerting & Production Failures

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Every symptom belongs to one component. Name the component, then check the one thing that confirms it.

⏱ 4 min · Topic 1 of 8

This is the on-call chapter. Eighteen chapters of architecture and authoring were building toward one practical skill: seeing a symptom and knowing which component to open first.

Start with the router below. Every row maps something you can observe to exactly one component — which is the entire return on understanding the architecture in Chapter 2.

It is 3am. Start from what you can see.
Component: Workers, pools or parallelism
  • Check first: Free pool slots, running-task count against parallelism, and whether any workers are alive on the right queue.
  • Then: Chapter 15’s ladder in order: pool, per-DAG limit, parallelism, workers. Never look upstream — queued means dependencies already passed.
Every row maps a symptom to exactly one component. That mapping is the whole value of Chapter 2 — without it, every incident starts with restarting everything.
Core mental model

Every symptom belongs to one component. Name the component, then check the one thing that confirms it.

Why it matters

Without this mapping, every Airflow incident starts by restarting everything and hoping. With it, most incidents are a single check. The difference is not knowledge of Airflow internals — it is having the symptom-to-component table before you need it.

Scheduler heartbeat
The signal that scheduling is alive. Stale means nothing is being scheduled, while the UI stays perfectly healthy.
Zombie task
A task whose worker stopped heartbeating past task_instance_heartbeat_timeout. The scheduler fails it — the task did not fail, the worker vanished.
Runbook
The symptom-to-action mapping written down before the incident. This chapter is one.
Common mistake

Restarting every component when something breaks. You lose the running tasks along with the problem, learn nothing about the cause, and it recurs next week. Scheduler-down and dag-processor-down look nothing alike once you know what to check.

Better habit

Read the task state before the logs. The state names the component (Chapter 3).

Keep a written runbook. Under pressure, recall is unreliable and a table is not.

After every incident, add the symptom you did not recognise to the runbook.

Architecture is an on-call skill

Chapter 2 was not background. It is the reason “nothing is scheduling” and “my fix did not apply” are two different investigations rather than one panicked restart.

Remember this

Symptom, component, check. That sequence is most of Airflow operations.