What You'll Master Here
A batch transform turns one bounded input set into one or more output sets plus evidence.
Pure Python batch transforms are the middle ground between tiny row helpers and full analytical engines. They fit when the input is small enough, bounded by chunking, or already sampled for a control-plane task.
This chapter teaches the patterns behind real batch jobs: map, filter, group, join, dedupe, aggregate, sorted scan, and window-like state.
The goal is not to avoid pandas or SQL. The goal is to know the core data movement so you can choose the right engine and still reason about correctness.
A batch transform turns one bounded input set into one or more output sets plus evidence.
Many pipeline bugs are pattern bugs: fanout joins, wrong dedupe winners, unreported rejects, and metrics computed at the wrong grain.
- bounded batch
- A dataset small enough to materialize safely or a deliberately limited chunk.
- business key
- The field or tuple that identifies the entity you are deduping or joining.
- grain
- What one output row represents, such as one customer, one day, or one customer-day.
raw batch
bounded input records
normalize
typed fields
classify
accepted / rejected
aggregate
metrics or outputs
report
counts reconcile
Using pure Python for an unbounded warehouse-scale join. The code may work on samples and fail when state grows.
State the input size assumption before materializing.
Name the output grain before aggregating.
Return rejected rows and summary counts alongside outputs.
I would use pure Python when the batch is bounded, then choose dictionaries for lookup joins, defaultdict for grouping, Counter for tallies, and sorted scans for latest-row logic.
Pure Python batch code is about explicit data movement, not clever loops.