What You'll Master Here
A data pipeline is plumbing for data — it moves data from where it is created to where it creates value, reshaping it on the way, automatically and repeatedly.
Most people meet "data pipelines" as a wall of tool names: Kafka, Airflow, Spark, dbt, Snowflake. This chapter ignores the tools and builds the one idea underneath all of them from first principles. If you can explain how water reaches a tap, you can understand a data pipeline.
The map below is this chapter drawn as the thing it teaches. Open any box to see the job it does, what goes wrong inside it, and which topic teaches it — then start wherever the map makes you curious rather than at the top.
A data pipeline is plumbing for data — it moves data from where it is created to where it creates value, reshaping it on the way, automatically and repeatedly.
Every later chapter — ingestion, orchestration, streaming, observability — is one box of this map seen up close. Without the map they land as unrelated trivia; with it, the syllabus organises itself.
- first principles
- Reasoning from what a thing must do and why, rather than from the names of the tools that do it; the method this chapter uses throughout.
- data flow
- The path data takes from the moment it is created to the moment someone acts on it; the thing you trace when reasoning about any data system.
- data-informed decision
- The end point of every pipeline: a human or a system acting on a number the pipeline produced. Data has no value until a decision uses it.
- end-to-end trace
- Following one real record all the way from its source to the decision it informs; the core diagnostic habit of a data engineer.
- Four stages, two guardrails. Data flows left to right through the four numbered boxes: it enters at 01, is written down at 02, is reshaped at 03, and is handed to a human or a model at 04.
- The two guardrails are not a fifth box. Nothing flows through them — they span all four stages. Orchestration decides when each box runs; observability decides whether you can believe what came out. A pipeline missing them still runs; it just fails without telling anyone, which is worse.
- One framing note so it does not trip you later: Chapter 2 re-cuts this same material as five stages by promoting observe into the spine. Both cuts are used in the field. This chapter holds the guardrails outside the four boxes on purpose, because that is the arrangement that shows they apply to every stage rather than following them.
Thinking a pipeline is "just a script that copies data". You ignore the hard parts — reliability, correctness, freshness — that are the reason the job exists.
Trace data from where it is born to where a human decides something with it.
For any system, ask "what is one row, and where did it come from?".
Treat tools as interchangeable implementations of the four stages.
A pipeline is not the tools. It is a promise: the right data, in the right shape, in the right place, fresh enough to trust, without a human doing it by hand.
Do not read it top to bottom on faith. Open the map, pick the question you cannot answer yet, and jump there. Almost every topic here is something to do, not something to skim.
A data pipeline moves data from where it is created to where it creates value, automatically and reliably; this chapter is the map the whole course builds on.
- In one sentence, define a data pipeline without naming any tool.
- Pick an app you used today and trace one data flow: where a row is born, and the decision it informs.