What You'll Master Here
Requirements and constraints first, then sources, then shape, then reliability. Tools last, and only after asking whether anything needs building at all.
This is the capstone. The previous twenty-three chapters gave you the building blocks; this one gives you the method for assembling them into a design for a problem nobody has solved for you yet.
An architect does not open with bricks. They ask how many people live here, whether anyone uses a wheelchair, what the budget is and when you need to move in — and the building follows from the answers. Nobody would trust one who started by describing their favourite cladding. Data design works exactly the same way, and the equivalent mistake is opening with Kafka.
This chapter is deliberately not a demonstration. You will make the decisions: ask a vague stakeholder real questions and get realistically evasive answers, judge whether a pipeline should exist at all, design Marlow’s platform decision by decision, then run the whole method again on a problem that is nothing like it.
Requirements and constraints first, then sources, then shape, then reliability. Tools last, and only after asking whether anything needs building at all.
Knowing concepts is necessary and not sufficient. The job is applying them under ambiguity, with a budget and a team you did not choose. A repeatable method is what turns "I understand pipelines" into "I designed the right one for this".
- requirements-first design
- Deriving the architecture from what the data must achieve, before naming a single tool.
- clarifying questions
- The questions that turn a vague request into concrete numbers. Four of them decide most of the architecture.
- constraints
- Budget, team size and skills, the existing stack, the deadline. Usually the strongest real influence on what gets built, and almost never written in the brief.
- the build gate
- Deciding whether a pipeline should exist at all. Half of the requests in this chapter are better served by a query, a connector or a dashboard connection.
- trade-off reasoning
- Justifying each decision by the requirement it serves and stating what it costs you.
Starting a design by naming tools. You commit to technology before understanding the problem, and build the wrong thing impressively.
Designing as though budget, team size and the existing stack were details. You produce an architecture nobody can afford to run or staff, which is a wrong design rather than a compromised one.
Open every design with questions, never with a diagram.
Derive each decision from a stated requirement or constraint.
Ask whether anything needs building before deciding what to build.
A pipeline design is a chain of justified decisions, each tracing back to a requirement or a constraint. Get those right and the architecture nearly designs itself; the tools are the last and least interesting choice.
Do not read it. Use it. Every topic from here has something you decide, and the two design labs near the end are the point of the whole chapter — the second one especially, because it is the one that tells you whether the method transferred.
Applied design is requirements-and-constraints-first: a repeatable method that asks whether to build at all, then derives the architecture from what the data must achieve, with tools chosen last.
- Explain why naming tools first is the wrong way to start a design.
- Name the two inputs to a design that are almost never written in the brief.