Data Observability & Monitoring

In the data quality chapter, you wrote checks. A check looks at one table, on one run, and says pass or fail.

Checks only find problems you thought of. This chapter shows how to watch your data over time and across many tables. It also shows how to tell the right person.

What you already know

• In a DAG, a failed task makes later tasks skip, and a person gets an alert.
• A check is a query. A failed check can warn, quarantine rows, or block the publish.
• A freshness check uses MAX(order_date). A volume check compares the row count with a recent average.
• In the volume example, the usual count was 200, the allowed band was 100 to 400, and 12 failed.

What you will learn

1. Monitoring and observability: two different questions.
2. Five signals to watch.
3. How to find what is normal, using a run log.
4. How to send alerts that people answer.
5. How to promise a deadline, called an SLO.
6. Lineage: which tables and dashboards break together.
7. One incident, step by step.

We use the shop. The six orders are real. Numbers over time are made up for a bigger shop, and we say so each time.

Monitoring asks: is the job running? Observability asks more: is the data right? If not, why not, and who is hit?

Here are the two side by side:

A run log is a table with one row per run. We use a run log for a bigger shop. Its numbers are made up.

Ask the log for the status of every run:

SELECT run_date, status, orders_loaded
FROM   run_log
ORDER  BY run_date;

Look at the last row. The run for 2026-02-07 loaded 12 orders, and the usual is about 200. The job did not fail, but the data is wrong.

Monitoring checks things you chose in advance. Observability gives you enough signals to find problems you did not expect.

A signal is one measurement that tells you if data is healthy. Observability uses five. Here they are, with a shop example for each:

You already know freshness and volume from the data quality chapter. The schema chapter covers how columns change. Distribution is new, so here it is.

Distribution means how the values in a column are spread. Two simple numbers describe it: the share of NULL values and the average. Run both on the real orders:

SELECT COUNT(*)                                   AS orders,
       SUM(CASE WHEN amount IS NULL THEN 1 ELSE 0 END) AS null_amounts,
       ROUND(100.0 * SUM(CASE WHEN amount IS NULL THEN 1 ELSE 0 END)
             / COUNT(*), 1)                            AS null_pct,
       ROUND(AVG(amount), 2)                           AS avg_amount
FROM   raw_orders;

Order 105 is pending, so a NULL amount is fine today. Say the NULLs are on pending orders, so every row passes your checks. But 60% pending is not normal, and revenue is far too low.

Four signals detect problems. Lineage does not detect anything. It tells you what a problem touches.

A signal needs a number to compare with. We call that number the baseline. It is what normal looks like.

The run log stores the signals for every run, so you can build a baseline from it. It also holds run time and NULL share.

This query finds the usual count from the first five days and checks the last day:

WITH usual AS (
  SELECT ROUND(AVG(orders_loaded), 1) AS avg_orders
  FROM   run_log
  WHERE  run_date BETWEEN '2026-02-02' AND '2026-02-06'
)
SELECT r.run_date, r.orders_loaded, u.avg_orders,
       CASE WHEN r.orders_loaded < 0.5 * u.avg_orders
              OR r.orders_loaded > 2   * u.avg_orders
            THEN 'ALERT' ELSE 'ok' END AS status
FROM   run_log r
CROSS  JOIN usual u
WHERE  r.run_date = '2026-02-07';

The same idea works for the other columns. The run took 3 minutes, and the usual is about 12. A much faster run often did less work.

A shop may have a weekly pattern, for example quiet Sundays. Then compare a Sunday with other Sundays, not with Friday.

An alert is a message sent when a signal looks wrong. A good alert goes to one person who can act, and it says what to do.

On-call means one person is on duty this week and answers urgent alerts. An urgent alert is called a page. It can wake that person up, so use it rarely.

Not every alert is equally urgent. Give each one a level. This table shows three levels:

On-call means one person is on duty this week and answers pages. A page wakes that person up, so use it rarely.

Alert fatigue means people get so many alerts that they stop reading them.

Here is a noisy rule. It pages when orders_loaded is below 200. Compare it with the band from the last section:

SELECT run_date, orders_loaded,
       CASE WHEN orders_loaded < 200 THEN 'page' ELSE 'ok' END AS noisy_rule,
       CASE WHEN orders_loaded < 100 OR orders_loaded > 400 THEN 'page' ELSE 'ok' END AS band_rule
FROM   run_log
ORDER  BY run_date;

With too many false pages, people miss a real alert, like 02-07. They did not need to try harder. They needed a rule that fits what normal looks like.

A small drop, such as 195 orders, can be a message or a ticket.

Give every page a runbook. A runbook is a short note that says what the alert means and the first things to check.

Your team needs a clear promise about the data. Three short terms help:

• SLI (service level indicator): a number you measure, such as the share of days daily_revenue was ready by 06:00.
• SLO (service level objective): the target for that number, such as 99% of days.
• SLA (service level agreement): a promise to other people, usually in a contract, with a penalty if you break it.

Teams often set the SLO stricter than the SLA, for example SLO 99% and SLA 95%. A miss then warns you before the promise to others is broken.

A publish log has one row per day, with the time the data was ready. We use ten made-up days. Measure the SLI with a query:

SELECT COUNT(*) AS days,
       SUM(CASE WHEN ready_at <= TIME '06:00' THEN 1 ELSE 0 END) AS on_time,
       ROUND(100.0 * SUM(CASE WHEN ready_at <= TIME '06:00' THEN 1 ELSE 0 END)
             / COUNT(*), 1) AS on_time_pct
FROM   publish_log;

The SLO says "daily_revenue ready by 06:00 on 99% of days". You measure an SLO over a time window, for example the last 100 days. 99% of 100 days is 99 days, so one late day is allowed.

Ten days is too short a window: one late day already drops the SLI to 90%.

Lineage is the map of what reads what. When one table breaks, the map shows which tables and dashboards break with it.

Here is the shop. orders is the app table. Look at the table of who reads what:

Downstream means the tables and dashboards that read from a table. Upstream means the tables it reads from. When raw_orders breaks, you look downstream to find who is affected.

Three assets are hit. daily_revenue and customer_report read raw_orders directly. revenue_dashboard reads daily_revenue, so it breaks too.

The owners of these assets are the people to tell.

You can also look upstream. If revenue_dashboard shows a wrong number, follow the "Reads from" column back: daily_revenue, then raw_orders, then orders. The cause is on that path.

Some tools map columns, not only tables. Say the column amount breaks. customer_report reads only order_id and customer_id from raw_orders. daily_revenue reads amount. So only daily_revenue and revenue_dashboard are affected, and customer_report is safe.

dbt draws lineage for the tables it builds.

OpenLineage is an open standard. Tools such as Airflow and Spark send lineage events in it. A tool such as Marquez draws the map.

A hand drawing goes out of date fast.

Now use everything on the made-up day 2026-02-07. The run succeeded, but it loaded 12 orders.

Follow the eight steps in order:

1. The volume check fails: 12 orders is outside the band. It is a page, because the dashboard must be right at 06:00.
2. The on-call engineer says "I have it". This is called acknowledging. The alert stops going to other people.
3. Look at the other signals. Freshness passes, the columns are the same, and the NULL share is 0%. With only 12 rows, 0 NULLs is normal. Only volume is wrong, so rows are missing.
4. Use lineage. raw_orders feeds daily_revenue, customer_report and revenue_dashboard. Tell their owners now.
5. Look upstream for the cause. Here, the export from the app was cut short.
6. Put a banner on the dashboard: "Today’s numbers are wrong. A fix is coming." Do this before the fix.
7. Fix the export. Run the job again from the extract_orders task, so all the orders load. The checks pass, and the data is published. The data is ready at 06:40, so this day counts as late in the SLI. One late day in 100 still meets the 99% SLO.
8. Add a check that compares the rows in the export with the rows in the app database.

Step 8 is the most valuable. Every incident should end with a new check, so the next one is found sooner.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.