What You'll Master Here
Quality is measurable across specific dimensions. Encode each as an automated test, run it at the last moment you can still change your mind, and decide in advance what happens when it fails.
A pipeline that moves data flawlessly but delivers wrong data is worse than no pipeline, because people trust it and act on it. This chapter is about data quality: systematically checking that the data is correct, complete and sane, and catching problems before they reach a dashboard or a decision.
A supermarket till is the everyday version. It scans and totals perfectly — but if one item is mislabelled at £2.40 instead of £24.00, the receipt is still added up flawlessly and still wrong. Nothing in the till is broken. The number it prints is simply not true, and the only way to find out is to check the items against what was really in the basket.
We start from a precise definition. "Quality" is not one thing; it is a set of measurable dimensions — completeness, validity, uniqueness, consistency, timeliness, accuracy — each answering a specific question about your data. Turning those dimensions into automated tests is what makes quality a property you can guarantee rather than hope for.
By the end you will be able to write validation checks for any important table, put them in the one place in the pipeline where they can actually stop harm, decide what to do when data fails them, and express the whole thing in dbt, Great Expectations or Soda.
Quality is measurable across specific dimensions. Encode each as an automated test, run it at the last moment you can still change your mind, and decide in advance what happens when it fails.
Bad data does not announce itself; it silently corrupts metrics, models and decisions. Data quality checks are the smoke detectors that turn invisible corruption into a loud, early failure, which is the entire point of trustworthy data engineering.
- data quality
- How well data meets expectations across dimensions like validity, completeness and accuracy.
- validation check
- An automated assertion about data (e.g. "amount is never negative"), run on every pipeline execution.
- upstream / downstream
- Upstream is everything that produced your data before you touched it (the shop database, an API, another team). Downstream is everyone who reads what you publish — dashboards, models, other pipelines. Data flows one way; blame flows the other.
- consumer
- Anyone or anything downstream that reads your table. A person with a dashboard, a machine-learning job, a finance report.
- serving table
- The table consumers actually query — the finished, published output of your pipeline, as opposed to the raw or intermediate copies only you look at. Teams often call these layers bronze (raw), silver (cleaned) and gold (serving).
- quarantine
- Routing failing rows aside for inspection instead of dropping them or letting them through.
- circuit breaker
- Halting a pipeline when quality checks fail badly, to stop bad data spreading. Named after the switch in a fuse box that cuts the power rather than let a fault burn the house down.
Assuming data is correct because the pipeline ran without errors. A successful run can still produce wrong data; without checks, the corruption ships unnoticed and gets used.
Define quality as concrete, measurable dimensions.
Encode every expectation as an automated test.
Decide the failure action (warn, quarantine, halt) up front.
You cannot trust what you do not test. Data quality is just making your assumptions about the data explicit and automatically enforced, on every run, so violations cannot slip through silently.
The next topic shows you ten real rows and asks you to find what is wrong with them, before a single term is defined. Do that first. Everything after it — the six dimensions, the SQL, the quarantine decision, the tools — is just a systematic way of doing what you will have done by hand.
Data quality is measurable across specific dimensions and enforced with automated checks; defining checks and failure actions is what turns trust from a hope into a guarantee.
- Explain why "the job succeeded" does not mean the data is correct.
- Name three things you would assert about an orders table before anyone is allowed to read it.