What You'll Master Here
capture every row, capture each row once, and be able to resume after a crash without losing or repeating anything.
Extraction is the "E" in ETL and ELT: the act of pulling data out of a source. It looks like the easiest step and is quietly the riskiest, because everything downstream is built on the assumption that extraction captured all the data, exactly once. If extraction drops or duplicates rows, no later stage can fix it.
This chapter is about the discipline that makes extraction trustworthy. You will learn the three guarantees every extractor must provide (completeness, no duplication, resumability), then the concrete techniques: pagination that never loses the tail, watermarks for incremental reads, rate limits and backoff, and extracting large sources in parallel.
By the end you will be able to write or review an extractor and immediately spot the bugs that silently lose data, the ones that pass every test on a small dataset and then drop rows in production. That instinct is what separates a reliable data engineer from someone who "wrote a script that worked once".
Extraction has one job stated three ways: capture every row, capture each row once, and be able to resume after a crash without losing or repeating anything.
Extraction failures are the most expensive kind because they are invisible and unrecoverable: a row never extracted is a row no downstream cleverness can recover. Getting extraction right is the foundation the entire pipeline’s correctness rests on.
- extraction
- Reading data out of a source (database, API, file, stream) into the pipeline.
- completeness
- The guarantee that extraction captured every row it should have.
- resumability
- The ability to continue an interrupted extraction without losing or duplicating data.
- checkpoint
- A saved position (a watermark or cursor) extraction can resume from after a failure.
Testing an extractor only on a small, well-behaved dataset. Pagination, rate-limit, and resume bugs stay hidden until production scale, where they silently drop rows.
Hold extraction to three guarantees: complete, no duplicates, resumable.
Assume the source will paginate, throttle, and fail mid-run.
Checkpoint progress so an interrupted extraction can resume safely.
Extraction is not "copy the data". It is "capture every row exactly once, and survive a crash mid-way". Hold those guarantees and the techniques in this chapter are just how you achieve them.
Read the guarantees first, then each technique as a way to satisfy them. The pagination topic is where most real extraction bugs live, study it closely.
Reliable extraction guarantees completeness, no duplication, and resumability; every technique in this chapter exists to satisfy those three promises under real-world failure.
- State the three guarantees a good extractor must provide.
- Explain why a row missed at extraction can never be recovered downstream.