Reliability, Failure & Recovery

Every pipeline fails sometimes. You cannot stop every failure. You can keep the damage small and make the recovery safe.

An incident is a failure that reaches the people who use the data.

After a failure, a senior engineer asks three questions:

• Who is affected?
• How do we find out before they do?
• How do we recover without a second incident?

What you already know

• Orchestration & Scheduling: retries, timeouts, and tasks that skip when an earlier task fails.
• Data Quality & Validation: checks, and write, audit, publish.
• Data Observability & Monitoring: alerts, service level objectives (SLOs), and lineage (the map of what reads what).
• Incremental Processing & Backfills: a backfill runs the job again for past days.
• Idempotency, Retries & Exactly-Once: safe re-runs.
• Storage Layers & File Formats: raw data, time travel and snapshots.

We do not teach those again. We use them to recover from a real failure.

What you will learn

1. Six ways a job fails.
2. How to limit the damage.
3. How to re-run and backfill safely.
4. How to replay, roll back and rebuild in order.
5. How to avoid a second incident.
6. How to set bad rows aside.
7. RPO and RTO: how much you can lose.
8. How to test the recovery plan.
9. The five steps of an incident, and a short review.

We use the shop. The six orders are real. Every incident is made up, and we say so each time.

Safe recovery rests on three things. This is only a recap. Each one was taught before:

• Idempotency: a re-run gives the same result (Idempotency, Retries & Exactly-Once).
• Isolation: a broken run does not touch what readers see (write, audit, publish in Data Quality & Validation).
• A known-good version: raw data kept as it arrived, and time travel or a snapshot (Storage Layers & File Formats).

If one is missing, some recovery steps in this chapter are not safe.

A failure is any run that does not give the right data. Some failures are loud: a task fails and an alert fires. Others are quiet: the job is green and the data is wrong.

Here are six failures, with the shop job as the example. Look at the table:

Retry a transient error. Do not retry a permanent error, because it fails the same way every time. Orchestration & Scheduling shows how to set retries and timeouts.

Partial failure is the one beginners miss. Say a rebuild of daily_revenue stops after two days.

The table daily_revenue_new holds only those two days (made up). Find the paid days that never arrived:

SELECT r.order_date
FROM   (SELECT DISTINCT order_date
        FROM   raw_orders
        WHERE  status = 'paid') r
LEFT   JOIN daily_revenue_new d
       ON d.order_date = r.order_date
WHERE  d.order_date IS NULL
ORDER  BY r.order_date;

If the job then publishes this table, the dashboard shows two days. Nobody notices that two more are missing.

Blast radius means everything one failure can reach. To contain a failure means to keep it from growing, and you do this first. Lineage, from Data Observability & Monitoring, tells you what is inside it.

Say raw_orders is wrong for 2026-01-21. Data Observability & Monitoring used lineage to find the three assets it reaches. Look at what you do with each:

Stop the two tables first, so the dashboard has nothing new to read. Then tell the three owners.

You have three tools to contain a failure:

• Stop downstream. Tasks that read the bad table do not run. A failed check already does this.
• Circuit breaker. After several failures in a row, stop calling the failing system for a while and page a person. Later, one test call checks if it is back.
• Mark data as not ready. Tell readers which days are safe to use.

Here is the third tool. A small table, table_status, has one row for each table. Its column ready_through is the last day that is safe to read.

Say 2026-01-21 is wrong (made up: 12.00), so ready_through stays at 2026-01-15:

SELECT d.order_date, d.revenue
FROM   daily_revenue d
JOIN   table_status s
       ON s.table_name = 'daily_revenue'
WHERE  d.order_date <= s.ready_through
ORDER  BY d.order_date;

To recover means to get back to correct data. Start with the smallest tool that works. Use this order:

1. Re-run the task that failed.
2. Backfill the days that are wrong.
3. Replay from the raw data.
4. Roll back the table to an older version.

The first two are here. The last two are in the next section.

Re-run one day

A re-run is safe only if the task is idempotent. That means a second run leaves the same result as the first (Idempotency, Retries & Exactly-Once). Here the task deletes its day, then inserts it again:

DELETE FROM daily_revenue
WHERE  order_date = '2026-01-21';

INSERT INTO daily_revenue
SELECT order_date, SUM(amount)
FROM   raw_orders
WHERE  status = 'paid'
  AND  order_date = '2026-01-21'
GROUP  BY order_date;

Backfill a range

A backfill is a new run for a past date. Use the six-step plan from Incremental Processing & Backfills. In a recovery, add two rules:

• Write down the range before you start, for example 2026-01-10 to 2026-01-21.
• Keep the old values until the new ones pass their checks.

Sometimes one day is not enough. The logic was wrong for many days. Then you replay.

Replay means you rebuild a table from the raw data. Raw data, also called bronze, is the first copy of the data, as it arrived.

You only add to it. You never change or delete its rows, so it is the truth you can rebuild from.

Say a bad build forgot the status filter (made up).

It counted the refunded order 103 and the pending order 105. So it made extra rows for 2026-01-11 (60.00) and 2026-01-20 (no amount). The fix filters to paid orders and rebuilds from raw_orders:

SELECT order_date, SUM(amount) AS revenue
FROM   raw_orders
WHERE  status = 'paid'
GROUP  BY order_date
ORDER  BY order_date;

Replay only works if the raw data is still there and still right. That is why a later section asks you to protect it.

Roll back a table

Sometimes the table itself is damaged, and a replay would take too long. Some table formats keep old versions.

Reading an old version is called time travel (Storage Layers & File Formats).

A rollback makes an old version the current one again. In Delta Lake it is RESTORE. In Apache Iceberg it is the rollback_to_snapshot procedure:

-- Delta Lake
RESTORE TABLE daily_revenue TO VERSION AS OF 12;

-- Apache Iceberg (Spark)
CALL catalog.system.rollback_to_snapshot('shop.daily_revenue', 4815162342);

You can only go back as far as old files are kept. VACUUM in Delta and expire snapshots in Iceberg delete them. By default Delta keeps deleted files for 7 days.

Rebuild in order

After you fix a table, fix what reads it. Follow lineage from upstream to downstream. Here is the order for the shop:

1. The source sends 2026-01-21 again. The new rows are added to raw_orders as a new batch, a group of rows loaded together. The rebuild reads only the newest batch for that day.
2. The daily_revenue and customer_report tables are rebuilt and checked. Both read raw_orders.
3. You move ready_through forward.
4. The revenue_dashboard refreshes. It reads daily_revenue through ready_through.

If you refresh the dashboard first, it reads the old table and shows the same wrong numbers.

Recovery work is done in a hurry, and a hurry makes mistakes. A second incident is a new failure that the fix itself causes. Two kinds are common.

The first is a backfill that adds rows instead of replacing them. Say 2026-01-21 was already right at 80.00.

A backfill adds it again with a plain INSERT. Look for days with more than one row:

SELECT order_date,
       COUNT(*)     AS rows_for_day,
       SUM(revenue) AS total
FROM   daily_revenue
GROUP  BY order_date
HAVING COUNT(*) > 1;

The second is recovering while people read the table. Without one transaction, a reader can see a day missing between the DELETE and the INSERT. Three habits help:

• Build the new data in a staging table, check it, then swap it in one step. This is write, audit, publish.
• Tell the readers before you start and when you finish.
• After the swap, run the same checks on the published table.

A poison row is a row the job cannot process as it is. A retry does not help, because the row stays the same. Stopping the whole load for one row hurts every reader.

So set the row aside. Data Quality & Validation showed the quarantine table: bad rows wait there while the good rows load. In streaming, the same idea is a dead-letter queue, a side queue for messages that keep failing.

Say a batch has three new orders (made up, ids from 140). The amount arrives as text, and one row has N/A.

This query finds the rows that do not look like an amount. It uses PostgreSQL:

SELECT order_id, amount_text
FROM   incoming_orders
WHERE  amount_text !~ '^[0-9]+[.][0-9]{2}$';

A NULL amount_text is not returned. A test on NULL gives NULL, not true, so WHERE drops the row. To set those rows aside too, add OR amount_text IS NULL.

A quarantine table needs three things:

• The row, exactly as it arrived.
• The reason it failed, and the time.
• Where it came from, such as the file or batch id, so you can load it again.

When the cause is fixed, load the quarantined rows again. They are safe to load again if the load is idempotent.

Two numbers set your recovery goals. RPO, the recovery point objective, is the most data you can lose, measured in time. RTO, the recovery time objective, is the most time you can take to be back.

Say daily_revenue is copied to a backup every night at 02:00 (made up). The table breaks at 15:00. The copy holds the table as of 02:00, so the last 13 hours of changes are not in it.

The worst case is a failure just before the next copy. So this backup can lose up to 24 hours.

It meets an RPO of 24 hours, but not of 1 hour. Look at three options, with made-up times:

Time travel is not a backup. It lives in the same storage, so deleting that storage, or running VACUUM, removes it too.

A replay is a recovery too, but it needs the raw data. So raw data is the one thing you must also copy somewhere safe.

Keep that copy where one mistake cannot delete both the raw data and the copy. A clean-up rule that deletes old files in one account can delete a copy that sits next to them.

Pick RPO and RTO from cost. Ask what one hour of lost data, or one hour of waiting, costs the business. Then choose the cheapest option that meets both numbers.

A plan nobody has run is only a document. You find the gaps when you try it, not when you read it. Two kinds of test help:

• A restore drill is a practice restore. Restore last night’s backup, or roll back one table, on a copy. Time it.
• A game day is a planned practice failure. Break one task on purpose on a copy of the pipeline. See who notices, and how.

After a restore drill, compare the restored table with the numbers you saved when you made the backup. Check the row counts and the totals, for example SUM(revenue). Matching counts alone can hide wrong values.

Then write down how long the restore really took. Say it took 3 hours (made up), but your RTO is 1 hour. The plan does not meet the goal, so change it.

A runbook is a short note of the steps to follow (Data Observability & Monitoring). Update it with what you learned, such as a missing permission or a wrong step. Run the drill again later, because systems change.

Data Observability & Monitoring walked through 2026-02-07 in eight steps. The run succeeded, but it loaded 12 orders, and the usual is about 200.

Here the same work falls into five steps. This chapter adds three parts: ready_through, the rebuild order and the written review.

Follow the five steps in order:

1. Detect. The volume check fails and pages the on-call engineer.
2. Contain. Stop daily_revenue and customer_report, and put a banner on the dashboard. Keep ready_through at 2026-02-06.
3. Fix. Find the cause. The export from the app was cut short, so fix the export.
4. Recover. Re-run from extract_orders and check the day. Rebuild the tables in order, move ready_through, then refresh the dashboard.
5. Review. Write down what happened, while people still remember it.

The review is the step that stops the next incident. A blameless review asks what in the system allowed the failure. It does not ask who to blame.

People tell the true story when they feel safe. A true timeline gives better actions. Use this short template, filled in with the made-up incident:

The action above finds the next failure sooner. Every review needs at least one action like it.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.