What You'll Master Here
who initiates (pull or push) and how much do we read (full or incremental)? Answer both and you have designed the ingestion.
Every pipeline begins the same way: getting data in. This is ingestion, and it is deceptively hard, because the source is the one part of the system you do not control. It can change shape, rate-limit you, go down, or quietly stop sending data, and your job is to capture everything anyway.
This chapter gives you the two mental maps that make ingestion tractable. First, the four families of sources (files, databases, APIs, event streams), because each family has a known pattern. Second, the two axes that describe any ingestion strategy: pull versus push (who initiates) and full versus incremental (how much you read).
By the end you will be able to look at any source and immediately know its family, whether to pull or be pushed to, whether to load it fully or incrementally, and which reliability traps (pagination, rate limits, retries, duplicates) you must handle. That is the practical core of being trusted with real data.
Ingestion has two questions: who initiates (pull or push) and how much do we read (full or incremental)? Answer both and you have designed the ingestion.
Ingestion is where most data is silently lost, because failures here are quiet: a skipped API page, a missed incremental window, a duplicate after a retry. Get ingestion right and everything downstream has a fighting chance; get it wrong and no later cleverness can recover the rows you never captured.
- ingestion
- Collecting raw data from a source into the pipeline, ideally unchanged.
- source
- Where data is born: a file, a database, an API, or an event stream.
- connector
- A reusable component that knows how to ingest from a specific source.
- watermark
- A bookmark of the last value processed, used to read only new data next time.
Treating ingestion as a trivial "copy the data" step. You ignore pagination, rate limits, and retries, and silently lose or duplicate rows in production.
Classify every source into one of the four families first.
Decide pull vs push and full vs incremental before writing code.
Assume the source will misbehave, and design for it.
You do not control the source, so ingestion is defensive engineering. Capture everything, exactly once, even when the source paginates, throttles, changes, or fails.
Learn the four families, then the two axes (pull/push, full/incremental), then the reliability traps. Together they let you design ingestion for almost any source you meet.
Ingestion is defensive capture from sources you do not control; master the four source families and the two axes (pull/push, full/incremental) and you can ingest almost anything.
- Name the four source families and the two ingestion axes.
- Explain why lost data at ingestion cannot be recovered downstream.