Pure Python Batch Transform Patterns

Openingloading the content…

Loadingone moment…

01 · Orientation

What You'll Master Here

A batch transform turns one bounded input set into one or more output sets plus evidence.

⏱ 4 min · Topic 1 of 15

Pure Python batch transforms are the middle ground between tiny row helpers and full analytical engines. They fit when the input is small enough, bounded by chunking, or already sampled for a control-plane task.

This chapter teaches the patterns behind real batch jobs: map, filter, group, join, dedupe, aggregate, sorted scan, and window-like state.

The goal is not to avoid pandas or SQL. The goal is to know the core data movement so you can choose the right engine and still reason about correctness.

Core mental model

A batch transform turns one bounded input set into one or more output sets plus evidence.

Why data engineers care

Many pipeline bugs are pattern bugs: fanout joins, wrong dedupe winners, unreported rejects, and metrics computed at the wrong grain.

bounded batch: A dataset small enough to materialize safely or a deliberately limited chunk.
business key: The field or tuple that identifies the entity you are deduping or joining.
grain: What one output row represents, such as one customer, one day, or one customer-day.

raw batch

bounded input records

normalize

typed fields

classify

accepted / rejected

aggregate

metrics or outputs

report

counts reconcile

Common mistake

Using pure Python for an unbounded warehouse-scale join. The code may work on samples and fail when state grows.

Better habit

State the input size assumption before materializing.

Name the output grain before aggregating.

Return rejected rows and summary counts alongside outputs.

What to say

I would use pure Python when the batch is bounded, then choose dictionaries for lookup joins, defaultdict for grouping, Counter for tallies, and sorted scans for latest-row logic.

Remember this

Pure Python batch code is about explicit data movement, not clever loops.

02 · Batch model

Batch Thinking: One Input Set To One Output Set

Rows seen should reconcile to accepted, rejected, and intentionally skipped rows.

⏱ 5 min · Topic 2 of 15

A batch transform starts with a finite population: a list, a chunk, a query result, or a small reference file.

The transform should make its contract visible: rows seen, accepted rows, rejected rows, and final output rows.

This is different from a streaming generator because you may sort, group, or inspect the whole bounded set.

Core mental model

Rows seen should reconcile to accepted, rejected, and intentionally skipped rows.

Why data engineers care

Batch jobs are easiest to audit when every input row has an explainable destination.

Normalize a bounded list and preserve rejectsworked example

Python

Input data

raw_orders3 rows

order_id	customer_id	amount	status
1001	c1	12.30	paid
1002	c2		paid
1003	c1	20.00	refunded

from decimal import Decimal

accepted = []
rejected = []

for raw in raw_orders:
    if raw["status"] == "refunded":
        rejected.append({"order_id": raw["order_id"], "reason": "refunded order excluded"})
        continue
    if not raw["amount"]:
        rejected.append({"order_id": raw["order_id"], "reason": "amount is required"})
        continue
    accepted.append({
        "order_id": raw["order_id"],
        "customer_id": raw["customer_id"],
        "amount": Decimal(raw["amount"]),
    })

Result · 3 rows

kind	order_id	detail
accepted	1001	amount Decimal("12.30")
rejected	1002	amount is required
rejected	1003	refunded order excluded

Common mistake

Dropping rows with continue and no rejected output. The batch no longer reconciles and downstream users cannot explain missing rows.

Better habit

Track rows_seen.

Make every exclusion visible.

Keep accepted output shape stable.

Audit habit

A batch transform should answer: how many rows arrived, how many were accepted, and exactly why the others did not move forward.

Remember this

Batch transforms should produce both data and an accounting trail.

03 · Core patterns

Map, Filter, And Projection Patterns

Map for shape, filter for population, project for contract.

⏱ 5 min · Topic 3 of 15

Map changes each row, filter decides which rows remain, and projection chooses the output fields.

These three operations cover a huge amount of transform code, but the order matters. Validate before projecting away evidence you might need for rejection reasons.

Keep transformation and filtering reasons visible so the output contract is reviewable.

Core mental model

Map for shape, filter for population, project for contract.

Why data engineers care

Most row-level Python transforms are combinations of these three ideas.

Filter invalid rows, then project the output contractworked example

Python

projected = [
    {
        "order_id": row["order_id"],
        "customer_id": row["customer_id"],
        "amount": row["amount"],
    }
    for row in accepted
    if row["amount"] > 0
]

Result · 1 row

order_id	customer_id	amount
1001	c1	12.30

Common mistake

Projecting fields before validation. The code may delete raw fields needed to explain a rejection.

Better habit

Validate with enough context.

Project only the final contract fields.

Keep field naming consistent across stages.

Readable pattern

If the comprehension becomes hard to read, switch to a for loop. Data code should be reviewable before it is compact.

Remember this

Map, filter, and project are simple, but the contract around them is where engineering happens.

04 · Grouping

Grouping With `defaultdict` And `Counter`

A grouping key is the output grain encoded as a dictionary key.

⏱ 5 min · Topic 4 of 15

Grouping creates buckets keyed by a value such as customer_id, status, or event_day. In pure Python, `defaultdict(list)` is the common bucket builder.

`Counter` is the right tool for tallies: statuses, reject reasons, event names, or file outcomes.

Grouping is where grain becomes concrete. A group keyed by customer_id produces one bucket per customer; a group keyed by event_day and status produces one bucket per day-status pair.

Core mental model

A grouping key is the output grain encoded as a dictionary key.

Why data engineers care

Wrong grouping keys produce plausible but wrong metrics.

Group orders and count statusesworked example

Python

from collections import Counter, defaultdict

orders_by_customer = defaultdict(list)
status_counts = Counter()

for order in accepted_orders:
    orders_by_customer[order["customer_id"]].append(order)
    status_counts[order["status"]] += 1

Result · 3 rows

output	key	value
orders_by_customer	c1	2 orders
orders_by_customer	c2	1 order
status_counts	paid	3

Common mistake

Grouping by display name instead of stable id. Renames split or merge buckets incorrectly.

Better habit

Use stable business keys.

Name tuple keys clearly.

Use Counter for counts instead of manual if-key-exists logic.

Standard library

collections.defaultdict supplies missing values from a factory; Counter is a dict subclass designed for tallies.

Remember this

Grouping is dictionary design plus grain discipline.

05 · Joins

In-Memory Joins With Lookup Dictionaries

Index the one-side by key, then enrich the many-side one row at a time.

⏱ 5 min · Topic 5 of 15

A lookup dictionary is the pure Python version of a dimension join. Build an index once, then attach reference fields to each fact row.

This avoids nested loops and makes missing reference rows explicit.

Before joining, confirm the lookup key is unique. If two customer rows share the same id, your dictionary will silently keep the last one unless you guard against it.

Core mental model

Index the one-side by key, then enrich the many-side one row at a time.

Why data engineers care

Lookup joins are fast and readable, but missing or duplicate keys can corrupt enrichment silently.

Join orders to customers with a lookup dictworked example

Python

Input data

customers2 rows

customer_id	country
c1	US
c2	CA

orders2 rows

order_id	customer_id	amount
1001	c1	12.30
1004	c9	7.00

customers_by_id = {customer["customer_id"]: customer for customer in customers}

joined = []
rejected = []
for order in orders:
    customer = customers_by_id.get(order["customer_id"])
    if customer is None:
        rejected.append({"order_id": order["order_id"], "reason": "missing customer"})
        continue
    joined.append({**order, "country": customer["country"]})

Result · 2 rows

kind	order_id	detail
joined	1001	country US
rejected	1004	missing customer

Common mistake

Building a lookup dict from non-unique keys. Later rows overwrite earlier rows and the join becomes non-deterministic by input order.

Better habit

Validate lookup uniqueness first.

Decide inner, left, or reject behavior for misses.

Keep missing-key rejects visible.

Fanout language

Say "I am indexing the one-side by unique key first" to show you are preventing nested-loop and fanout mistakes.

Remember this

A lookup dict is a join. Treat key uniqueness and missing matches as part of the contract.

06 · Dedupe

Dedupe By Business Key

For each key, run a tournament and keep the winner according to a documented rule.

⏱ 5 min · Topic 6 of 15

Dedupe keeps one winner per business key. The most important part is the tie-break rule, not the code.

Common winners are latest updated_at, highest version, latest ingestion_time, or a source-priority order.

A deterministic tie-break prevents reruns from choosing different winners when input order changes.

Core mental model

For each key, run a tournament and keep the winner according to a documented rule.

Why data engineers care

Duplicate source rows can double metrics or cause unsafe upserts.

Keep latest customer update by keyworked example

Python

latest_by_customer = {}

for row in customer_updates:
    key = row["customer_id"]
    current = latest_by_customer.get(key)
    if current is None or row["updated_at"] > current["updated_at"]:
        latest_by_customer[key] = row

deduped = list(latest_by_customer.values())

Result · 2 rows

customer_id	updated_at	tier
c1	2026-06-19T10:00:00	gold
c2	2026-06-19T08:00:00	silver

Common mistake

Using set(rows) or first row wins without a business rule. The result depends on incidental input order instead of source truth.

Better habit

Name the business key.

Name the winner rule.

Handle exact ties explicitly.

Upsert safety

Before a merge/upsert, staging data should have one deterministic row per business key.

Remember this

Dedupe is a business decision implemented with a dictionary.

07 · Metrics

Aggregations And Metric Reports

Metric key plus accumulator equals one output row per metric grain.

⏱ 5 min · Topic 7 of 15

Aggregation reduces many rows into metric rows. The grain could be day, customer, country, status, or a tuple of fields.

Use Decimal for money-like sums and initialize metric containers deliberately.

A metric report should include enough counts to validate the denominator and exclusions.

Core mental model

Metric key plus accumulator equals one output row per metric grain.

Why data engineers care

Most business decisions consume aggregated data, so grain and exclusions must be defensible.

Compute revenue by day and statusworked example

Python

from collections import defaultdict
from decimal import Decimal

revenue = defaultdict(Decimal)

for order in accepted_orders:
    key = (order["event_day"], order["status"])
    revenue[key] += order["amount"]

metric_rows = [
    {"event_day": day, "status": status, "revenue": amount}
    for (day, status), amount in sorted(revenue.items())
]

Result · 2 rows

event_day	status	revenue
2026-06-19	paid	32.30
2026-06-20	paid	18.00

Common mistake

Summing floats for money. Reconciliation can drift because binary floating point is approximate.

Better habit

Use Decimal for money.

Use tuple keys for multi-field grain.

Sort output rows for deterministic tests.

Test habit

Expected metric rows should be sorted and small enough to inspect by hand in a unit test.

Remember this

Aggregations need a named grain, exact numeric policy, and deterministic output ordering.

08 · Scans

Sorted Scans And Latest-Row Logic

Sort into the order you wish the answer could be read, then scan once.

⏱ 5 min · Topic 8 of 15

Sorting turns many stateful questions into one pass: latest row per key, first event per user, gap between events, or current status.

The sort key is part of the contract. For latest row per user, sort by user_id and event_time; for deterministic ties, add event_id or ingestion_time.

Once sorted, the scan can keep only the current key state instead of a huge structure, when the question allows it.

Core mental model

Sort into the order you wish the answer could be read, then scan once.

Why data engineers care

Latest-row bugs often come from missing tie-breakers or comparing timestamps as unnormalized strings.

Latest event per user after sortingworked example

Python

events_sorted = sorted(
    events,
    key=lambda row: (row["user_id"], row["event_time"], row["event_id"]),
)

latest_by_user = {}
for event in events_sorted:
    latest_by_user[event["user_id"]] = event

Result · 2 rows

user_id	event_id	event_time
u1	e2	2026-06-19T09:05:00+00:00
u2	e3	2026-06-19T08:00:00+00:00

Common mistake

Sorting only by timestamp and not by key. Latest per user becomes latest globally or depends on later dictionary overwrites.

Better habit

Normalize timestamps before sorting.

Include deterministic tie-breakers.

Sort output for stable tests.

What to say

I would sort by entity key and time, then scan to keep the latest row per entity with a deterministic tie-break.

Remember this

Sorted scans are pure Python window logic with explicit ordering.

09 · Ordering

Sorting, Ranking & Deterministic Tie-Breaks

Sort keys are tuples. Put the tie-break in the tuple; do not rely on the order rows happened to arrive in.

⏱ 6 min · Topic 9 of 15

Python's sort is stable — records that compare equal keep the order they arrived in — and that is the property people accidentally depend on. Stability is only useful if you know what order they arrived in, and rows from a set, a dict scan or a parallel read arrive in no order you chose.

So a sort with a tie is not deterministic unless you made it so. Add the tie-break column explicitly, and the same input produces the same output on every run, on every machine, in every Python version.

For "the top N" specifically, `heapq.nlargest` does the job without sorting the whole collection, which matters when the collection is large and N is small.

Core mental model

Sort keys are tuples. Put the tie-break in the tuple; do not rely on the order rows happened to arrive in.

Why data engineers care

A report whose rows swap places between runs looks broken even when every number is right, and a paginated result with a non-deterministic sort can show one row on two pages and another on none.

stable sort: Equal elements keep their relative input order. Python's sort is stable; the input order may not be meaningful.
tie-break: An extra key component that makes the order total, so no two rows compare equal.
heapq.nlargest: Top N by a key without sorting the whole collection.

A tie the sort cannot resolveworked example

Python

rows = [
    {"buyer": "eve", "amount": 120, "id": 3},
    {"buyer": "amir", "amount": 120, "id": 1},
    {"buyer": "cara", "amount": 90, "id": 2},
]

# One key: the two 120s keep their INPUT order, whatever that happened to be.
print([r["buyer"] for r in sorted(rows, key=lambda r: -r["amount"])])

# Tie-break in the key: the same answer on every run, from any input order.
print([r["buyer"] for r in sorted(rows, key=lambda r: (-r["amount"], r["id"]))])

Result · 2 rows

output
['eve', 'amir', 'cara']
['amir', 'eve', 'cara']

Both are "correct". Only the second is reproducible, because the first is relying on the order the rows were built in — which nothing in the pipeline guarantees.

Top N without sorting everythingworked example

Python

import heapq

rows = [
    {"buyer": "eve", "amount": 120, "id": 3},
    {"buyer": "amir", "amount": 120, "id": 1},
    {"buyer": "cara", "amount": 90, "id": 2},
]

# Highest amount first; lowest id wins a tie. Both directions in one tuple.
top = heapq.nlargest(2, rows, key=lambda r: (r["amount"], -r["id"]))
print([r["buyer"] for r in top])

Result · 1 row

output
['amir', 'eve']

`nlargest` keeps a heap of N rather than sorting the whole list, so the cost is O(n log N) instead of O(n log n). The negation flips one component of the key without reversing the other.

Choosing how to order

Need	Use	Cost
Whole collection ordered	`sorted(rows, key=...)`	O(n log n)
Top or bottom N, N small	`heapq.nlargest` / `nsmallest`	O(n log N)
Single max or min	`max(rows, key=...)`	O(n)
Mixed directions, numbers	Negate the component: `(-a, b)`	Free
Mixed directions, strings	Two passes, or `reverse` on the outer sort	Two sorts — strings cannot be negated

Common mistake

Sorting on one key and relying on stability for the rest. The output changes when the upstream read order changes — a different file order, a parallel read, a dict that was rebuilt. Nothing errors and the diff looks like a data change.

Reaching for `reverse=True` when only one component should be descending. It reverses the whole key, so the tie-break flips as well and ties come back in the wrong order.

Sorting the whole collection to take the top ten. On a large collection that is most of the runtime, and `heapq.nlargest` gives the same answer for a fraction of it.

Better habit

Make every sort key a tuple ending in something unique.

Negate a numeric component rather than reversing the whole sort.

Reach for `max`, `nlargest` or `nsmallest` when you do not need the full order.

Interview note

Adding a tie-break unprompted — "I would order by amount then id, so the result is deterministic" — is a small sentence that marks you as somebody who has debugged a report that changed for no reason.

Remember this

A sort with a tie is only reproducible if the key makes the order total. Put the tie-break in the tuple, and use a heap when you only want the top few.

10 · Windows

Window-Like Patterns In Pure Python

Partition by key, order rows, then carry just enough state for the window result.

⏱ 5 min · Topic 10 of 15

Some SQL window ideas translate cleanly to pure Python: row numbers, running totals, lag values, and first/latest rows.

The cost is that you own the ordering and state. SQL does this declaratively; Python requires you to sort, group, and carry state.

Use these patterns for bounded data and for learning. Move to SQL or Spark when partitions or state are large.

Core mental model

Partition by key, order rows, then carry just enough state for the window result.

Why data engineers care

Window-like requirements appear in interviews, reconciliation scripts, and small control-plane jobs.

Running total per customerworked example

Python

from collections import defaultdict

running_revenue = defaultdict(Decimal)
rows = []

for order in sorted(accepted_orders, key=lambda row: (row["customer_id"], row["event_time"])):
    running_revenue[order["customer_id"]] += order["amount"]
    rows.append({
        "customer_id": order["customer_id"],
        "order_id": order["order_id"],
        "running_revenue": running_revenue[order["customer_id"]],
    })

Result · 3 rows

customer_id	order_id	running_revenue
c1	1001	12.30
c1	1003	32.30
c2	1002	18.00

Common mistake

Computing running totals before sorting. The same rows produce different answers when input order changes.

Better habit

Sort before window-like scans.

Name partition and order keys.

Estimate state size per partition.

Engine boundary

For large partitions, prefer SQL window functions or Spark; pure Python is best for bounded batches.

Remember this

Window-like Python code is explicit partitioning, ordering, and state.

11 · Quality

Stateful Quality Checks

Use sets, counters, and lookup maps as quality sensors around the transform.

⏱ 4 min · Topic 11 of 15

Batch transforms are a natural place to check duplicates, missing references, impossible amounts, or unexpected status values.

Some checks are row-level; others need batch state, such as duplicate ids or totals by file.

Quality checks should produce structured evidence, not just print warnings.

Core mental model

Use sets, counters, and lookup maps as quality sensors around the transform.

Why data engineers care

Bad data that passes through a transform becomes harder to trace later.

Detect duplicate order idsworked example

Python

from collections import Counter

order_id_counts = Counter(row["order_id"] for row in raw_orders)
duplicate_order_ids = [
    {"order_id": order_id, "count": count}
    for order_id, count in order_id_counts.items()
    if count > 1
]

Result · 1 row

order_id	count
1001	2

Common mistake

Checking only the final output for duplicates. Rows may already have been overwritten or dropped before the check sees them.

Better habit

Check keys before dedupe when duplicates matter.

Return quality check rows.

Decide warning vs rejection vs fatal for each rule.

Reusable pattern

Counter is excellent for duplicate-key checks because it tells you both which key and how many times it appeared.

Remember this

Stateful checks turn hidden batch assumptions into inspectable evidence.

13 · Checklist

Interview And Production Checklist

Population, key, grain, state, output, evidence.

⏱ 4 min · Topic 13 of 15

A strong batch-transform answer states the grain, the key, the population, the failure path, and the output contract.

Use the smallest readable structure that proves the idea: dict for lookup, defaultdict for groups, Counter for tallies, sorted scan for latest/running logic.

Always finish by saying how you would test the output with small input and expected rows.

Core mental model

Population, key, grain, state, output, evidence.

Why data engineers care

Good batch code is easy to inspect, test, and rerun.

Batch transform summary contractworked example

Python

summary = {
    "rows_seen": len(raw_orders),
    "accepted": len(accepted),
    "rejected": len(rejected),
    "metric_rows": len(metric_rows),
}

Result · 1 row

rows_seen	accepted	rejected	metric_rows
5	3	2	2

Common mistake

Showing only code and not the expected output rows. The interviewer or reviewer cannot verify the grain or key logic.

Better habit

Use tiny input/output examples.

Sort outputs for deterministic tests.

Explain when to switch engines.

What to say

I would build the lookup once, validate key uniqueness, aggregate at the stated grain, and return both output rows and reconciliation counts.

Remember this

Batch transform competence is pattern fluency plus explicit contracts.

14 · Practice

Practice Lab

Predict what will go wrong before you run it, then check whether you were right about which line caused it.

⏱ 6 min · Topic 14 of 15

Four exercises for this chapter, in the module workspace. Each one runs against graded cases, and each has a trap that is in the data rather than in the algorithm.

Four exercises that are the batch shapes themselves: group, join, rank and reshape.

Do them with the chapter closed. If one goes wrong, come back to the section it belongs to rather than re-reading the whole thing.

Core mental model

Predict what will go wrong before you run it, then check whether you were right about which line caused it.

Why data engineers care

Reading about a failure mode and meeting one are different memories. The second is the one that is still there under interview pressure.

Common mistake

Opening the reference solution before your own version runs. You learn what correct looks like without learning what wrong feels like, and the wrong version is the one you will write first under pressure.

Better habit

Write the contract of the function — what goes in, what comes out, what happens to a bad row — before the first line.

Run the empty-input case early. It is the one most solutions forget and the one every grader checks.

Study tip

Run each solution twice in the same process on two different inputs. Anything that behaves differently the second time is carrying state it should not.

Remember this

The chapter tells you what to watch for. These four are where you find out whether you would have.

Active recall Lab problems

1Running total, restarting per keyA running total that restarts per key, in one ordered pass.Open 2Rank rows with a deterministic tie-breakRank rows on three keys with mixed directions, so the same input always ranks the same way.Open 3Pivot a long metrics table to widePivot a long metrics table to wide, deciding what an absent cell becomes.Open 4Top N per key, in bounded memoryKeep the top n rows per key with a heap, so memory is bounded by n rather than by the feed.Open

Pure Python Batch Transform Patterns

What You'll Master Here

Batch Thinking: One Input Set To One Output Set

Map, Filter, And Projection Patterns

Grouping With `defaultdict` And `Counter`

In-Memory Joins With Lookup Dictionaries

Dedupe By Business Key

Aggregations And Metric Reports

Sorted Scans And Latest-Row Logic

Sorting, Ranking & Deterministic Tie-Breaks

Window-Like Patterns In Pure Python

Stateful Quality Checks

When Pure Python Stops Being The Right Tool

Interview And Production Checklist

Practice Lab

Next Chapter

Pure Python Batch Transform Patterns

What You'll Master Here

Batch Thinking: One Input Set To One Output Set

Map, Filter, And Projection Patterns

Grouping With `defaultdict` And `Counter`

In-Memory Joins With Lookup Dictionaries

Dedupe By Business Key

Aggregations And Metric Reports

Sorted Scans And Latest-Row Logic

Sorting, Ranking & Deterministic Tie-Breaks

Window-Like Patterns In Pure Python

Stateful Quality Checks

When Pure Python Stops Being The Right Tool

Interview And Production Checklist

Practice Lab

Next Chapter