Data Pipeline

Source to sink, what breaks without a pipeline, and the four-stage, two-guardrail frame every later chapter zooms into.

Ingest, store, transform, serve, and observe: the end-to-end mental model and where bounded vs unbounded data fits.

Batch and streaming are two execution models, not two speeds: one runs a job to completion, the other never stops, and micro-batch just shrinks the batch window; choose between them by how fresh the answer actually needs to be.

Transform-then-load vs load-then-transform: the order decides which engine runs the transform and what form the data is in when it crosses into the destination — not reprocessability, who can build, or schema enforcement — plus the compliance/audit angle and how open table formats are dissolving the boundary.

Files, APIs, databases, events, and SaaS connectors; full vs incremental loads; pull vs push and where each fits.

Log-based, query-based, and trigger-based CDC; streaming changes off an operational DB, and why CDC beats polling.

Pagination, rate limits, watermarks and high-water marks, incremental keys, and auth, so extraction never silently drops rows.

Cleaning, deduplication, joins, enrichment, and type casting; row vs set-based logic and pushing work to the engine.

Append, upsert/merge, overwrite, and SCD loads; idempotent writes; bulk vs streaming inserts and their failure modes.

DAGs, dependencies, triggers, and backfills with Airflow, Dagster, or Prefect: cron vs event-driven scheduling.

At-most, at-least, and exactly-once semantics; safe reruns and dedupe keys, the hardest correctness problem in pipelines.

Partitioned reprocessing, replays, late-arriving data, and time travel, so you can reprocess without double-counting.

Producers, brokers, consumers, topics, partitions, and offsets: the Kafka / Kinesis / Pub-Sub mental model.

Windowing, watermarks, event-time vs processing-time, and stateful operators in Flink, Spark, and Kafka Streams.

Partitioning for order, consumer groups, checkpointing, dead-letter queues, and exactly-once inside a stream.

Batch + stream hybrids vs stream-only architectures, when to unify the two paths, and how to reprocess in streaming.

The six quality dimensions, a real check for each, where the gate belongs in the pipeline, quarantine vs circuit breakers, and the same rules written in dbt, Great Expectations and Soda.

The five pillars, what "normal" actually means (baselines and seasonality), SLI vs SLO vs SLA with error budgets, alert routing that survives contact with a rota, and lineage as a blast radius.

What a schema states in five languages, which side to deploy first per compatibility mode, expand-and-contract for breaking changes, registries and what they cannot protect, real contract files, and the semantic break that passes every check.

What bytes scanned actually cost, row vs columnar layout and encodings, inside a Parquet file, a lake built up from bucket to catalogued table, how a table format commit is atomic, and copy-on-write vs merge-on-read.

What a cluster is made of, sizing partitions with a number rather than a hunch, what a shuffle physically does, fixing skew with real code, reading a slow job off the Spark UI, and what every fix is worth in pounds.

Eight failure modes and which announce themselves, retry policies that help rather than hurt, what a checkpoint really guarantees, RPO and RTO priced against a real data-loss incident, restore drills, and the post-mortem that stops it twice.

Controller vs processor and what the acronyms mean, a treatment per column, why pseudonymised data is still personal data, who can actually call Decrypt, real row and column policies, and tracing one erasure request through twelve systems.

A six-step method including the build gate and the constraints nobody writes down, a stakeholder interview you conduct, two design labs you decide yourself, and a word-for-word interview narration with the pushback questions.

Every named technology on the pipeline canvas — 72 entries across five families — with what it is, when to reach for it, pros, cons, how the bill is shaped and what it costs, plus five costed reference stacks.