What You'll Master Here
Assume every component fails. Contain the failure, make every re-do safe, know what you can afford to lose, and write down what you learned.
Everything in a distributed pipeline fails eventually: machines crash, networks drop, sources go down, someone merges a config change on a Friday afternoon. Reliability is not the absence of failure — that is not on offer — but correctness despite it, and clean recovery when a part does go down.
An aeroplane is the everyday version. It has two engines when one would fly, a checklist for every failure anyone has thought of, and crews who rehearse engine failure in a simulator twice a year. None of that means aeroplanes are fragile. It is precisely why they are dependable, and it is the posture this chapter asks you to take toward a pipeline.
By the end you will be able to name the failure modes and say which ones announce themselves, write a retry policy that helps rather than hurts, explain exactly what a checkpoint guarantees and what it does not, put a price on losing data, and write the post-mortem that stops the same failure twice.
Assume every component fails. Contain the failure, make every re-do safe, know what you can afford to lose, and write down what you learned.
Failures are guaranteed; the only question is whether your pipeline handles them gracefully or corrupts data and pages someone in a panic. This is what lets a business trust data continuously, not just when everything happens to be working.
- reliability
- Continuing to operate correctly despite component failures. Not "it never breaks" — "when it breaks, nothing is corrupted and it comes back cleanly".
- recovery
- Returning to a correct state after a failure without losing or duplicating data.
- silent failure
- A failure that raises no error while producing wrong or incomplete data. Six of the eight failure modes in this chapter are silent, and they are the expensive ones.
- graceful degradation
- Continuing partial, useful operation instead of collapsing — publishing yesterday’s data with a banner rather than nothing at all.
- disaster recovery (DR)
- The plan for the failures that are not routine: a region down, an account compromised, a dataset deleted.
- blast radius
- Everything a given failure affects. Introduced with lineage in Chapter 18; here it is the thing your DR copy must sit outside of.
Designing only for the happy path. The first crash, network blip or source outage either corrupts data or takes the pipeline down hard, and both are discovered by someone downstream.
Treating reliability as separate from observability. You cannot recover from a failure you never detected, and most failures do not announce themselves. Chapter 18 is half of this chapter’s answer.
Assume every component fails, and design the containment before the feature.
Make every re-do safe, so recovery is never itself a risk.
Write down what a failure cost, and what changed because of it.
Reliability is not "it never breaks". It is "when it breaks, nothing is corrupted, it recovers cleanly, and a human can fix it fast". Engineer for the failure, not just the success.
The next topic recaps the three earlier ideas everything here depends on — idempotency, checkpoints and dead-letter queues. Ten minutes there and the rest of the chapter is mechanics rather than mystery.
Reliability also has to be measured and staffed, but that half — SLI versus SLO versus SLA, error budgets, alert severity and the on-call rota — is owned by the observability chapter, in far more depth than a recap here could manage. This chapter covers what happens when something breaks; that one covers how you find out and who is told.
SLIs, SLOs and alerting: Chapter 18Reliability is correctness despite inevitable failure: contain it, make every re-do safe, know what you can afford to lose, and turn each incident into a permanent change.
- Explain why "it never fails" is the wrong reliability goal.
- Name three earlier concepts this chapter depends on.