What You'll Master Here
Polling asks the database "what changed?" over and over. CDC lets the database announce every change as it happens. Push, not poll, and nothing slips through.
In Chapter 5 you learned incremental loads driven by a watermark: read the rows where updated_at is newer than last time. It works, but it has a quiet flaw, it cannot see deletes, and it can miss updates that change a row without bumping its timestamp. Change Data Capture (CDC) is the robust answer.
CDC means capturing every change a database makes, inserts, updates, and deletes, as a stream of events, the moment they happen. Instead of repeatedly asking the database "what looks new?", you tap into the database’s own record of what changed and let it tell you, in order, with nothing missed.
By the end of this chapter you will understand the three ways to do CDC, why log-based CDC is the gold standard at scale, how deletes (the thing polling forgets) are handled, and how tools like Debezium turn a database’s private change log into a public stream that powers real-time pipelines at Uber, Airbnb, and Netflix.
Polling asks the database "what changed?" over and over. CDC lets the database announce every change as it happens. Push, not poll, and nothing slips through.
CDC is how modern companies keep warehouses, search indexes, caches, and microservices in sync with their operational databases in near real time, without hammering those databases with constant queries. It is one of the most important ingestion techniques in the field.
- Change Data Capture (CDC)
- Capturing every insert, update, and delete in a database as a stream of change events.
- transaction log
- The database’s own append-only record of every change (Postgres WAL, MySQL binlog).
- change event
- A message describing one row change: the operation, the key, and before/after values.
- Debezium
- A popular open-source log-based CDC tool that streams database changes into Kafka.
Assuming a polling-by-timestamp approach captures everything. Deletes are invisible and some updates slip past, so the target slowly drifts out of sync with the source.
Reach for CDC when you need deletes and every update reliably.
Prefer letting the database announce changes over repeatedly polling it.
Think of a database’s change log as a stream waiting to be read.
Every database already keeps a perfect, ordered log of everything that changed, for its own crash recovery. CDC just reads that log. You are not inventing change tracking; you are tapping a record that already exists.
Start with why polling falls short, then the three CDC methods, then log-based CDC in depth. The deletes topic is the one that most often appears in interviews.
CDC streams every insert, update, and delete from a database as it happens; it fixes the blind spots (deletes, silent updates) that timestamp polling cannot see.
- Explain one change a timestamp-based incremental load would miss.
- In one sentence, say what a transaction log is and why CDC reads it.