Data Pipeline

What a pipeline is, why people build one instead of copying by hand, the five stages from source to serve, and why a re-run must give the same result.

Ingest, store, transform, serve, and observe: the end-to-end mental model and where bounded vs unbounded data fits.

Batch and streaming are two execution models, not two speeds: one runs a job to completion, the other never stops, and micro-batch just shrinks the batch window; choose between them by how fresh the answer actually needs to be.

Transform-then-load vs load-then-transform: the order decides which engine runs the transform and what form the data is in when it crosses into the destination — not reprocessability, who can build, or schema enforcement — plus the compliance/audit angle and how open table formats are dissolving the boundary.

Files, APIs, databases, events, and SaaS connectors; full vs incremental loads; pull vs push and where each fits.

Log-based, query-based, and trigger-based CDC; streaming changes off an operational DB, and why CDC beats polling.

Pagination, rate limits, watermarks and high-water marks, incremental keys, and auth, so extraction never silently drops rows.

Cleaning, deduplication, joins, enrichment, and type casting; row vs set-based logic and pushing work to the engine.

Append, upsert/merge, overwrite, and SCD loads; idempotent writes; bulk vs streaming inserts and their failure modes.

DAGs, dependencies, triggers, and backfills with Airflow, Dagster, or Prefect: cron vs event-driven scheduling.

At-most, at-least, and exactly-once semantics; safe reruns and dedupe keys, the hardest correctness problem in pipelines.

Partitioned reprocessing, replays, late-arriving data, and time travel, so you can reprocess without double-counting.

Producers, brokers, consumers, topics, partitions, and offsets: the Kafka / Kinesis / Pub-Sub mental model.

Windowing, watermarks, event-time vs processing-time, and stateful operators in Flink, Spark, and Kafka Streams.

Partitioning for order, consumer groups, checkpointing, dead-letter queues, and exactly-once inside a stream.

Batch + stream hybrids vs stream-only architectures, when to unify the two paths, and how to reprocess in streaming.

Copies and the ISR, what acks=all really waits for, choosing the key and the partition count, rebalances during a deploy, compaction, Kafka Connect, a schema rollout, replay by time, and Kafka against Kinesis, Pub/Sub and Event Hubs.

How a Flink job runs, where state lives, checkpoint barriers, restarting after a crash, savepoints for upgrades, two-phase commit into Kafka, backpressure, and when Spark is enough.

The four questions that choose a serving store, when the warehouse is enough, rollups before serving, Pinot, Druid and ClickHouse, updates and repeats in a fast store, late events, and what freshness costs.

The six quality dimensions, a real check for each, where the gate belongs in the pipeline, quarantine vs circuit breakers, and the same rules written in dbt, Great Expectations and Soda.

The five pillars, what "normal" actually means (baselines and seasonality), SLI vs SLO vs SLA with error budgets, alert routing that survives contact with a rota, and lineage as a blast radius.

What a schema states in five languages, which side to deploy first per compatibility mode, expand-and-contract for breaking changes, registries and what they cannot protect, real contract files, and the semantic break that passes every check.

What bytes scanned actually cost, row vs columnar layout and encodings, inside a Parquet file, a lake built up from bucket to catalogued table, how a table format commit is atomic, and copy-on-write vs merge-on-read.

Inside an Iceberg table, time travel without copying, two writers at once, hidden partitioning and partition evolution, the upkeep jobs, inside a Hudi table, catalogs shared by many engines, and when a warehouse is better.

What a cluster is made of, sizing partitions with a number rather than a hunch, what a shuffle physically does, fixing skew with real code, reading a slow job off the Spark UI, and what every fix is worth in pounds.

Eight failure modes and which announce themselves, retry policies that help rather than hurt, what a checkpoint really guarantees, RPO and RTO priced against a real data-loss incident, restore drills, and the post-mortem that stops it twice.

Controller vs processor and what the acronyms mean, a treatment per column, why pseudonymised data is still personal data, who can actually call Decrypt, roles with one masked, row-filtered view, and tracing one erasure request through every copy.

A six-step method including the build gate and the constraints nobody writes down, a stakeholder interview you conduct, two design labs you decide yourself, and a word-for-word interview narration with the pushback questions.

The real tools for each box on the pipeline canvas, on AWS, Google Cloud, Azure and open source. What each one does, how its bill is shaped, and five reference stacks.