PRODUCTION FLOWSLifecycle

The Data Pipeline Lifecycle

How data moves in production — pick a topic on the left and its full breakdown loads here: the mental model, real pipeline diagrams and worked examples, failure modes, and the habits that keep data flowing correctly and on time.

⏱ 18 min readTopics chapter readerLevel · Foundations
01 · Orientation

What You'll Master Here

A pipeline’s lifecycle is a loop, not a line — ingest, store, transform, serve, watched throughout by observe, repeating on every schedule.

⏱ 5 min · Topic 1 of 11

Chapter 1 gave you four stages wrapped in two guardrails. This chapter walks the full lifecycle a piece of data lives through — ingest, store, transform, serve — and promotes the fourth guardrail, observe, to a stage of its own, with a name, an owner and a section.

The widget below assembles a real report, an Uber "trips this week" city-operations review, one concrete sentence per stage. Reorder it, then send Observe to the one slot it actually belongs in.

By the end you should be able to take any data product and narrate every stage it passed through, naming the choice an engineer made at each one. That narration is the core skill of a data engineer.

Core mental model

A pipeline’s lifecycle is a loop, not a line — ingest, store, transform, serve, watched throughout by observe, repeating on every schedule.

Why it matters

Tools change every few years; the lifecycle does not. An engineer who thinks in these five stages picks up any new platform quickly, because every platform is a different implementation of the same five steps.

lifecycle
The full journey data takes through a pipeline — ingest, store, transform, serve, observe — repeated on every run.
stage
One named step of the lifecycle with a single clear job; naming the stage is the first move in debugging.
Uber, a Monday-morning “trips this week” reportFive stages, five concrete sentences. Reorder the spine, then send Observe to where it actually belongs.not checked yet
Before you touch anythingChapter 1’s company picker used Uber too, and called it streaming — surge pricing recomputes a multiplier per geofence inside a latency budget. This chapter uses Uber again and calls it batch. Both are true at the same time, in the same company, over data from the same trips. “Is Uber batch or streaming?” is not a question with an answer, because batch and streaming are properties of a pipeline, not of a company, and a company that size runs hundreds of both. The question that does have an answer is always about one named data product — what decision does this report serve, and how fresh must it be for that decision? Surge pricing is worthless if the number is a minute old; a weekly city-operations review is not harmed by a number that is a day old. Topic 8 turns that observation into the distinction that actually matters.Chapter 1 drew four stages wrapped in two guardrails and held observability outside the boxes on purpose, to show that it applies to every stage rather than following them. This chapter promotes observe to stage status so it has a name, an owner and a section of its own — and promotion means elevation, not insertion: observe still does not sit between serve and anything, and nothing flows through it. Orchestration remains a guardrail in both drawings.
The spine — move each card up or down
  1. position 1IngestEvery completed trip is copied out of the trips service's database once an hour into our landing zone, exactly as the service recorded it: trip_id, rider_id, driver_id, request time, pickup time, drop-off time, distance, fare, currency, city.
  2. position 2Transformsilver.trips is filtered to the seven days of the reporting week, cancelled and test trips are removed, fares are converted to a single currency, and the result is grouped into gold.trips_weekly — one row per city per week.
  3. position 3StoreThose hourly copies are appended to bronze/trips/dt=2026-06-15/ and never edited; a de-duplicated, one-row-per-completed-trip table is built from them into silver.trips.
  4. position 4Servegold.trips_weekly is what the Monday-morning city-operations dashboard reads, and the same table is exported to the finance team's model.
  5. position 5 — a decoy slotObserveEvery run checks that the week's trip count is inside a stated band of the previous four weeks, that no fare_usd is NULL, and that gold.trips_weekly advanced its week_end — and pages a human when any of the three fails.
Spans all four — the correct place for ObserveEmpty. Observe is still sitting in the spine above, at position 5.
Every wrong ordering here teaches something specific — check whenever you like.
Common mistake

Treating the five stages as a one-way line that runs once. You design steps that break the second time they execute, and reprocessing becomes a rewrite instead of a re-run.

Better habit

For any data product, name all five stages it passed through before naming a tool.

Ask what decision each stage makes, not just what runs there.

Assume every step runs again tomorrow, and design for repetition.

Promotion, not insertion

Chapter 1 held observe outside the four boxes to show it applies to every stage rather than following them. This chapter gives it a name and a section — but it still does not sit between serve and anything. Two drawings of one system, not a contradiction.

Remember this

A pipeline’s lifecycle is five repeating stages — ingest, store, transform, serve, observe — and mastering the stages, not the tools, is what makes the skill durable.

Practice1 prompt
  1. Narrate a weekly report through the five stages, and say why Observe spans them rather than sitting after Serve.