What You'll Master Here
A schema is a promise to whoever reads your data. Compatibility says whether a change keeps the promise; a contract makes the promise explicit, owned and enforced.
Chapter 18 ended with an unfixed root cause. Marlow’s mobile team started sending a new value in the channel field, the data team’s filter silently discarded it, and seven days of revenue went missing. Nobody did anything wrong. The two teams had simply never agreed what that field was allowed to contain. This chapter is that agreement.
Think of a wall socket. Its shape is a promise between the wall and every appliance ever plugged into it. Adding a USB port beside it breaks nothing. Changing the pin spacing breaks every appliance in the country on the same morning — which is why, when countries really do change a plug standard, they fit both kinds for years first.
Schemas work the same way, and two ideas do the heavy lifting. Compatibility tells you whether a change is safe in each direction. Data contracts write the promise down and make it enforceable.
A schema is a promise to whoever reads your data. Compatibility says whether a change keeps the promise; a contract makes the promise explicit, owned and enforced.
Schema changes are one of the most common causes of pipeline breakage, and they get worse as more teams depend on each other’s data. Compatibility rules and contracts are what let an organisation evolve its data without constant outages.
- schema
- The stated shape of a piece of data: which fields exist, what type each one is, whether it may be absent, and which values are legal. Not just a list of column names.
- producer
- Whoever creates the data — the shop service emitting order events, the team owning the source table. In this chapter the producer is the one making the change.
- consumer
- Anything that reads it: a Kafka consumer, a dbt model, a SQL filter, a dashboard, a machine-learning job. Producers routinely do not know who all of theirs are.
- schema evolution
- Changing that shape over time while keeping existing consumers working.
- compatibility
- Whether a specific change lets old and new code still read each other’s data. It has a direction, and the direction is what everyone gets wrong.
- schema registry
- A service that stores every version of a schema and refuses to accept a new one that would break consumers.
- data contract
- An explicit, owned, enforced agreement covering the schema plus the things a schema cannot say — what fields mean, how fresh they are, and how they will change.
Changing a produced schema without considering who reads it. Downstream consumers break unexpectedly, and the producer often did not know they existed.
Assuming a schema is just column names and types. The two things that actually cause outages — which values are allowed, and what the numbers mean — live outside that definition and get changed casually.
Treat any consumed schema as a promise you must not casually break.
Classify every change by compatibility before shipping it.
Make producer-consumer expectations explicit as contracts.
The moment someone reads your data, its shape is a contract — whether or not anyone wrote one. Evolving it safely means changing it in compatible ways, or running a migration, never silently breaking the promise.
Start with what a schema actually is in five different systems, then the direction rules, then how to ship a breaking change anyway, then enforcement, then contracts. The last topic is the one to remember: a change that passes every gate in this chapter and is still wrong.
Schemas are promises to consumers; compatibility rules and data contracts let producers evolve them safely instead of breaking everyone downstream.
- Explain why a schema becomes a contract the moment it is consumed.
- Name a schema change that is safe and one that is breaking.