What You'll Master Here
Testing checks known rules; observability watches for anything abnormal. Observability’s job is to make silent failures loud and early — and to tell exactly one person, once.
Data quality (Chapter 17) checks the rules you knew to write. But pipelines fail in ways nobody anticipated: a job that silently stops, a source that sends half its usual rows, a column whose values quietly shift. Observability is how you see those — the failures no specific test was written for.
A smoke alarm is the everyday version of a test: it knows exactly one thing and shouts about it. Observability is closer to noticing that the house feels colder than usual, that the meter is spinning faster than it should, that a room you never go into has been dark for a month. Nothing has triggered an alarm. Something is still wrong.
The distinction matters. Testing asks "does the data break a rule I defined?". Observability asks "is anything about this data different from normal?". You need both, because the incidents that hurt most are the ones nobody thought to test for.
By the end you will be able to instrument a pipeline — that is, add the measurements that let you see inside it while it runs — for all five pillars; define what "normal" means without pager-storming your team every Sunday; separate SLI from SLO from SLA; route alerts so they stay trustworthy; and use lineage to work out who to tell.
Testing checks known rules; observability watches for anything abnormal. Observability’s job is to make silent failures loud and early — and to tell exactly one person, once.
The worst data incidents are the silent ones a test never anticipated: a feed that stopped, a number that drifted. Observability is what turns those invisible failures into early alerts, and it is what lets a team trust data they did not personally verify today.
- data observability
- Continuously measuring the health of data itself — how fresh, how much, what shape, what structure — so unexpected problems become visible without anyone having predicted them.
- silent failure
- A failure that produces no error anywhere: the job succeeds, the tests pass, and the data is still wrong. The entire reason this chapter exists.
- to instrument
- To add measurements to something so you can see inside it while it runs. A pipeline is instrumented when every important dataset emits its own freshness, row count and value statistics on every run.
- blast radius
- Everything downstream of a failure that is therefore suspect — the dashboards, models and systems that read the broken data. Covered in full in the lineage topic.
- SLI, SLO, SLA
- Three different things people say "SLA" for. SLI is the measurement, SLO is the target you hold yourself to, SLA is the promise you make to someone else — with a consequence if you break it. Taken apart properly in the alerting topic.
Relying only on explicit tests and ignoring abnormality detection. Failures nobody wrote a test for — a stopped feed, a drifting value, a whole segment filtered out — go completely unnoticed.
Monitoring the job instead of the data. A green Airflow DAG proves the code ran. It says nothing about whether the numbers it produced are right, and most silent failures happen inside a successful run.
Pair rule-based testing with abnormality-based observability.
Monitor the five pillars on every important dataset, not just the ones you built.
Track lineage so the impact of any failure is always answerable.
Tests catch the problems you predicted; observability catches the ones you did not. Mature pipelines run both, because the failures that hurt most are usually the ones nobody thought to test for.
Start with the next topic — fourteen days of a real dashboard where nothing fails and everything is wrong. Once you have seen that, the five pillars stop being a list to memorise and become the answer to a question you are already asking.
Observability measures data health to catch unexpected problems, where tests only catch predicted ones; its five pillars plus baselines, alerting and lineage turn silent failures into early, traceable, actionable alerts.
- Explain the difference between a data-quality test and observability in one sentence each.
- Give an example of a failure a test would miss but observability would catch.