Monitoring, Logging, Alerting & Production Failures

Your shop runs one DAG every day. Its name is shop_orders_daily.

A DAG is a set of tasks with an order. This one has two tasks.

The first, load_orders, copies the orders. The second, build_daily_revenue, builds the daily_revenue table.

Pipelines break. In this chapter you answer three questions:

1. Where do you read what a task did?
2. How do you get told, without too many alerts?
3. What do you do first when a run goes wrong?

You know two things from earlier chapters. Data Observability & Monitoring taught freshness, volume and lineage. Reliability, Failure & Recovery taught failure modes, retries, backfills and runbooks.

This chapter shows the Airflow side of the same work.

A paid order counts as revenue. Refunded and pending orders do not. Every example uses these orders:

Each run reads one day, called its data interval, as in Scheduling, Data Intervals & the Logical Date Trap. The run about 2026-01-20 reads order 105 and starts on 2026-01-21 00:00. The log lines, times and messages in this chapter are made up.

A task log is the text a task writes while it runs. Airflow keeps one log for each try of each task. You open it from the grid view in the UI.

Look at the log of load_orders for the run about 2026-01-20:

[2026-01-21, 00:00:04 UTC] INFO - Starting attempt 1 of 3
[2026-01-21, 00:00:05 UTC] INFO - Reading orders from 2026-01-20 up to 2026-01-21
[2026-01-21, 00:00:06 UTC] INFO - Rows read: 1
[2026-01-21, 00:00:06 UTC] WARNING - Order 105 has no amount
[2026-01-21, 00:00:07 UTC] INFO - Rows loaded: 1
[2026-01-21, 00:00:07 UTC] INFO - Marking task as SUCCESS

The WARNING is the line to read. Order 105 is pending, so its amount is NULL. The task still passes, because a pending order has no amount yet.

Airflow has five levels:

By default, a log file stays on the machine that ran the task. On Kubernetes, that machine is a pod. A pod is deleted when its task ends, and the log goes with it.

With Celery workers, a replaced worker takes its logs too. Remote logging fixes both.

It copies each log to object storage, which is cheap file storage in the cloud. Look at the settings:

# airflow.cfg
[logging]
remote_logging = True
remote_base_log_folder = s3://shop-airflow-logs/tasks
remote_log_conn_id = aws_logs

Many failures are short. The network drops, or the database is busy. A retry runs the same task again.

You set retries in default_args. These values apply to every task in the DAG:

from datetime import timedelta

default_args = {
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "execution_timeout": timedelta(minutes=20),
    # "retry_exponential_backoff": True,  # optional: the wait grows
}

Now see a short problem. The database refuses two connections, then answers. The times are made up:

A retry helps only with short problems. A bug in your code fails all three tries. A retry is also safe only when the task is idempotent: running it twice gives the same result.

For example, build_daily_revenue first deletes the rows of its day, then inserts them again. The chapters Idempotency & Safe Task Design and Backfills, Catchup, Retries & Deadlines go deeper.

You met the parts of Airflow in Airflow Architecture: Scheduler, Executor, Workers, Metadata DB. The scheduler decides what runs and when.

A worker runs the task. Each symptom points to one part.

Five words first:

• A heartbeat is a regular "I am alive" signal that a part sends to the database.
• A zombie task shows as running, but its worker is gone. Airflow sees no heartbeat and fails it.
• A pool is a set of slots. A task needs one free slot to run.
• A sensor is a task that waits for something, such as a file, to appear.
• A deferred task waits without a worker slot. The triggerer part watches for its event.

Look at the symptom, the likely cause, and the first thing to check:

Here is the line a task log shows when the system kills a task. The line is made up:

[2026-01-21, 00:03:41 UTC] ERROR - Task exited with return code -9

A metric is a number that Airflow measures over time, such as how many tasks are queued. In the table, a page wakes a person now, and a ticket waits for the next working day.

Airflow can send metrics to StatsD or OpenTelemetry. Both are tools that collect numbers. You turn them on in the [metrics] section of the config.

These metrics measure Airflow itself. Data Observability & Monitoring covered metrics about your data, such as freshness and volume. Look at six metrics that warn you first:

Airflow also has a health endpoint. It says if the database is healthy, and when the scheduler and triggerer last sent a heartbeat. A simple check can read it every minute.

An alert is a message that tells a person something is wrong. Alerts fail in two ways.

Too few, and users find the problem first. Too many, and people mute the channel. Alert fatigue means that people stop reading alerts.

Airflow calls a callback when a task changes state. A callback is a function that you write.

An SLA is the time a task should finish by, counted from the start of its run. Look at the four callbacks:

The setting email_on_failure sends an email when a task fails. It needs a mail server (SMTP). A chat message is usually seen faster.

Here is a callback that sends a chat message. Set it in default_args:

def notify_failure(context):
    ti = context["task_instance"]
    start = context["data_interval_start"]
    end = context["data_interval_end"]
    message = (
        f"FAILED: {ti.dag_id}.{ti.task_id}\n"
        f"Data interval: {start:%Y-%m-%d} up to {end:%Y-%m-%d}\n"
        f"Log: {ti.log_url}"
    )
    send_to_chat(message)  # your own function

default_args = {"on_failure_callback": notify_failure}

This is the message it would send if load_orders failed for the run about 2026-01-20:

FAILED: shop_orders_daily.load_orders
Data interval: 2026-01-20 up to 2026-01-21
Log: https://airflow.example.com/...

You met pages and tickets in the metrics table. Only page for what a person must fix now.

Alert on what users feel: late data or wrong data. Do not alert on every cause. Look at six events:

Give every page an action. If the person cannot act, make it a chart or a ticket.

Some failures raise no error. The task is green, but the data is wrong. This is a silent failure, as in the Data Quality & Validation chapter.

Airflow cannot see a silent failure. Only a check on the data can. Look at four cases and the check for each:

Take the first case. The run about 2026-01-15 loaded nothing, and the task is still green.

This query compares two counts. The orders_loaded table is empty:

SELECT
  (SELECT COUNT(*) FROM orders
   WHERE order_date >= '2026-01-15' AND order_date < '2026-01-16') AS source_orders,
  (SELECT COUNT(*) FROM orders_loaded
   WHERE order_date >= '2026-01-15' AND order_date < '2026-01-16') AS loaded_orders;

A zero is not always wrong. Nothing was ordered on 2026-01-12, so both counts are 0 and the check passes. The task fails only when the two counts differ:

def check_load(source_orders, loaded_orders, day):
    if source_orders != loaded_orders:
        raise ValueError(f"{day}: loaded {loaded_orders} of {source_orders} orders")

Triage means deciding what to check first. A fixed order saves you from guessing while you are tired.

Start with the task state. The state names the part to open. Look at six states:

Then follow six steps, in this order:

1. Scope: is it one DAG or all of them? All of them points to the platform.
2. State: read the task states from the table above.
3. First failure: open the first failed task, not the last red one.
4. Code: check that the deployed code is the version you expect.
5. Stop the damage: pause the DAG if it writes wrong data.
6. Repair: clear the task to rerun the same run, or backfill runs that do not exist.

Step 3 matters most. Every task after a failed task gets upstream_failed, and its log holds nothing useful. The cause is in the first one.

This story is made up. Use its shape for your own incident story. A good story has five parts:

1. What happened.
2. The impact.
3. How you found it.
4. The fix.
5. The follow-ups.

What happened

The shop team changed its app on 2026-01-21. New orders now save the status as PAID, in capital letters. Old orders keep paid.

Our build_daily_revenue task counts rows where status is paid. The warehouse compares text exactly, so PAID is not paid.

Order 270 is made up. Look at the run about 2026-01-21:

SELECT DATE '2026-01-21' AS revenue_date, COALESCE(SUM(amount), 0) AS revenue
FROM orders
WHERE status = 'paid'
  AND order_date >= '2026-01-21'
  AND order_date <  '2026-01-22';

The impact

The dashboard said 80.00 for 2026-01-21. The right number was 115.00, which is 80.00 plus 35.00. Finance reads this number every morning.

Only one run was wrong. The next run, about 2026-01-22, starts on 2026-01-23 00:00. We fixed the rule before then.

How we found it

We did not find it. A finance analyst did. She compared the dashboard with the payment report.

That was the real failure. No alert fired, because nothing failed. The times are made up:

The fix

First I told people. At 10:15 I wrote to finance that the number for 2026-01-21 was wrong.

I said the right number is 115.00 and a fix is coming.

Then I changed the query to compare lowercase text. Look at the new query:

SELECT DATE '2026-01-21' AS revenue_date, COALESCE(SUM(amount), 0) AS revenue
FROM orders
WHERE LOWER(status) = 'paid'
  AND order_date >= '2026-01-21'
  AND order_date <  '2026-01-22';

At 10:30 I cleared the run about 2026-01-21. A clear reruns the run.

Deleting the day before inserting makes the rerun safe. The dashboard then said 115.00.

The follow-ups

A postmortem is a short written review after an incident. A blameless one looks at the system, not at a person. I did four things so this cannot happen quietly again:

1. I added a check that fails the task when a status is not a known value. After the fix, PAID is a known value too.
2. I sent that failure to the page channel, with the data interval and the log link.
3. I asked the shop team to tell us before they change a value. Each side has one owner.
4. I wrote a blameless postmortem and added the symptom to the runbook.

Here is the check. A NULL status is never in a list, so the check tests it on its own. It lists orders with a status we do not know:

SELECT id, status, amount
FROM orders
WHERE (status NOT IN ('paid', 'refunded', 'pending') OR status IS NULL)
  AND order_date >= '2026-01-21'
  AND order_date <  '2026-01-22';

The result: one wrong day, fixed before the next run. The check now fails the task as soon as a new status appears.

Seven things to remember:

• Read the task state first. It names the part to open.
• Save logs outside the worker or pod. A deleted log cannot come back.
• Retries fix short problems. Set retries, retry_delay and execution_timeout.
• Page only for what a person must fix now. Count retries; do not page them.
• Watch the health of Airflow itself: scheduler and triggerer heartbeats.
• A green run can hold wrong data. Add checks on the data inside the task.
• Tell an incident in order: what happened, impact, how found, fix, follow-ups.

Check yourself:

1. Tasks stay scheduled or queued, and nothing fails. What do you check?
2. Why does a failed-task alert miss a dead scheduler?
3. A run is green, but daily_revenue for 2026-01-15 shows 0.00. What do you check first?

Answers:

1. Scheduled means the task waits for a pool slot. Queued means it waits for a worker. Check the pool, then the workers.
2. No tasks run, so none fail. Alert on the scheduler heartbeat instead.
3. Compare the orders read with the orders in the source. Order 104 should be there.