Process data that does not fit in memory using lazy evaluation, yield, itertools, and streaming file pipelines.
⏱ 22 min readTopics chapter readerLevel · Intermediate
01 · Orientation
What You'll Master Here
A stream is a promise to produce the next item on demand, not a container holding every item now.
⏱ 4 min · Topic 1 of 16
Many beginner Python examples load everything into a list. Data engineering work often cannot do that: files are large, events arrive continuously, and downstream writers prefer fixed-size batches.
Iterators and generators let Python process one record at a time. They are the foundation for streaming file ingestion, memory-stable transforms, chunked writes, and pipeline code that does not collapse when data grows.
This chapter teaches the mental model behind lazy processing, when it helps, when it hurts, and how to keep evidence such as line numbers and rejected rows while streaming.
Core mental model
A stream is a promise to produce the next item on demand, not a container holding every item now.
Why data engineers care
A job that works on 100 rows but loads 10 million rows into memory is not production-ready. Streaming patterns keep memory predictable.
iterable
An object that can produce an iterator, such as a list, file handle, dict, or generator.
iterator
An object that returns the next value when next() is called and eventually raises StopIteration.
generator
A function or expression that yields values lazily while preserving local state.
batch
A bounded group of records processed or written together.
file handle
one line ready
parser
one dict
validator
accepted or rejected
normalizer
one typed row
writer
flush batch
Common mistake
Converting every reader to list(reader) before processing. Memory usage grows with input size and the job fails when files get large.
Better habit
Stream records until you have a reason to materialize them.
Keep source line numbers attached before validation.
Batch writes deliberately instead of buffering the whole dataset.
What to say
I would design the job as a lazy pipeline: read one record, parse it, validate it, normalize it, then write in bounded batches so memory stays predictable.
Remember this
Streaming Python is not magic. It is disciplined one-record-at-a-time processing with explicit batching.
02 · Motivation
Why Iteration Matters In Data Engineering
Move the cursor forward through the source and attach evidence as each record passes.
⏱ 5 min · Topic 2 of 16
Data engineering pipelines often sit between storage systems. The goal is not to own all data in memory; the goal is to move records through parsing, validation, enrichment, and writing safely.
Iteration gives you a stable contract with large input: ask for one item, process it, then ask for the next. That means row-level errors can be captured without keeping the full file in RAM.
Lazy iteration also improves observability. You can count rows seen, accepted, rejected, and written as the stream moves.
Core mental model
Move the cursor forward through the source and attach evidence as each record passes.
Why data engineers care
Memory-stable jobs scale more gracefully and fail closer to the bad record instead of failing after loading an entire dataset.
List loading versus streamingworked example
Python
# Risky for large files: stores every row before work starts.rows=list(read_ndjson(path))# Memory-stable: handles each row as it arrives.forrowinread_ndjson(path):handle(row)
Result · 2 rows
pattern
starts work when
memory behavior
list(read_ndjson(path))
after full file load
grows with file size
for row in read_ndjson(path)
first row
roughly one row plus buffers
Common mistake
Materializing data just to count or inspect it. The job may double memory usage before useful work starts.
Better habit
Count while streaming when possible.
Sample a small prefix with islice instead of loading all records.
Materialize only when an algorithm truly needs random access.
Operational signal
If memory rises with every input row, the job is probably collecting records accidentally. A streaming design keeps memory mostly flat.
Remember this
Iteration lets a job start doing useful work before the source is fully loaded.
03 · Core model
Iterables, Iterators, And One-Pass Consumption
An iterator is a cursor. Moving it forward changes what future code can see.
⏱ 5 min · Topic 3 of 16
An iterable can be looped over. An iterator is the object that remembers where the loop currently is. Calling iter(...) creates or returns an iterator; calling next(...) asks for the next value.
Many iterators are one-pass. Once consumed, they do not restart unless you create a new iterator from the source.
This matters because debugging code that "peeks" at a stream can accidentally consume the first records before the real transform sees them.
Core mental model
An iterator is a cursor. Moving it forward changes what future code can see.
Why data engineers care
One-pass behavior can create quiet missing-row bugs if a stream is inspected, counted, or reused incorrectly.
iter(...), next(...), and one-pass stateworked example
The iterator was consumed. The final list is empty because the cursor is already exhausted.
Common mistake
Counting an iterator with len(list(rows)) and then trying to process rows. The processing step sees an exhausted iterator.
Better habit
Treat iterators as one-pass unless you know otherwise.
Sample streams with care and pass the sampled rows onward if needed.
Create a fresh reader if you truly need a second pass.
Debugging trap
Printing list(iterator) is destructive for one-pass iterators. It is fine in a scratch cell, but dangerous inside reusable code.
Remember this
An iterator is not a list. Once you advance it, that value is gone from the stream.
04 · Generators
Generators And `yield`
yield means "return this value now, remember where I was, and continue later."
⏱ 5 min · Topic 4 of 16
A generator function uses yield to produce values lazily. When called, it returns a generator object; the function body runs only when the caller asks for the next item.
After yielding a value, the generator pauses with its local variables preserved. On the next next() call, it resumes after the yield.
This makes generators ideal for pipeline stages: parse raw records, yield valid records, and keep moving without building a giant list.
Core mental model
yield means "return this value now, remember where I was, and continue later."
Why data engineers care
Generators let you express record-by-record logic clearly while keeping memory bounded.
Yield accepted records as they are parsedworked example
The representative reader is def read_ndjson(path: Path) -> Iterator[dict[str, object]].
Common mistake
Using return record inside a generator loop. The generator stops after the first record instead of yielding the whole stream.
Better habit
Use yield for many outputs over time.
Use return only to stop the generator.
Annotate generator outputs with Iterator[...] when the function yields a stream.
Python behavior
A generator function does not execute immediately when called. It executes as the generator is consumed.
Remember this
Generators make one-record-at-a-time pipeline stages readable.
05 · Files
Streaming Files Line By Line
parse it, accept it, reject it, then move on.
⏱ 5 min · Topic 5 of 16
Text files and NDJSON files are naturally streamable. A file handle can be looped line by line, which keeps memory stable and preserves line numbers for errors.
The key habit is to attach source evidence before validation: file name, line number, raw payload, and parse error.
A malformed line should not necessarily destroy the whole stream. For NDJSON, you can reject the bad line and continue with later lines.
Core mental model
Each line is a mini boundary: parse it, accept it, reject it, then move on.
Why data engineers care
Line-numbered rejected rows let operators fix bad payloads without guessing which part of a large file failed.
Parse NDJSON with line-numbered rejectsworked example
Parsing the full file as one JSON document when it is NDJSON. One bad line or many valid lines fail as one document instead of record-by-record.
Better habit
Use enumerate(handle, start=1) for line evidence.
Catch parse errors at the row boundary when the format supports it.
Keep accepted and rejected outputs structurally different or clearly labeled.
Failure path
Streaming does not mean ignoring errors. It means each record can produce accepted output or rejected evidence without loading the full file.
Remember this
Line-by-line streaming gives you memory safety and better error evidence.
06 · Pipelines
Generator Pipelines
each stage touches one record and passes it forward.
⏱ 5 min · Topic 6 of 16
A generator pipeline connects small lazy stages: read raw lines, parse JSON, validate payloads, normalize records, and write batches.
Each stage receives an iterable and yields another iterable. The whole chain stays lazy because no stage needs the full dataset at once.
This style also keeps responsibilities small. read_ndjson does parsing; validate_events decides validity; normalize_events creates output rows.
Core mental model
A generator pipeline is an assembly line: each stage touches one record and passes it forward.
Why data engineers care
Lazy stages let you compose readable pipeline code without sacrificing memory safety.
Parse -> validate -> normalize as lazy stagesworked example
Python
defonly_accepted(events):foreventinevents:ifevent.get("kind")=="accepted":yieldeventdefrequire_user_id(events):foreventinevents:ifevent.get("user_id"):yieldeventelse:yield{"kind":"rejected","line_number":event["line_number"],"reason":"user_id is required",}defnormalized_events(events):foreventinevents:ifevent.get("kind")=="accepted":yield{"event_id":event["event_id"],"user_id":event["user_id"],"line_number":event["line_number"],}
Result · 3 rows
stage
input
output
stream_events
file lines
accepted or rejected parse results
require_user_id
parsed events
validated events or rejects
normalized_events
valid events
warehouse-ready rows
Common mistake
Making each stage return a list because it feels simpler. The pipeline silently stops being streaming and buffers every stage.
Better habit
Use iterables as inputs and yields as outputs for streaming stages.
Keep one responsibility per generator.
Name stages by their data contract.
Design check
If every stage can be tested with a three-row list, but can also consume a file stream, the boundary is probably right.
Remember this
Generator pipelines let you keep code modular and memory stable at the same time.
07 · Batches
Chunking And Batching
Materialize only the current batch, write it, clear it, repeat.
⏱ 5 min · Topic 7 of 16
Streaming one record at a time is memory-safe, but output systems often work better in batches. You might insert 500 rows at a time, write NDJSON chunks, or upload partition files.
Chunking groups a stream into fixed-size lists. The chunk is bounded, so memory remains predictable.
The final batch is often smaller than the requested batch size. Good batch code handles that without dropping it.
Core mental model
Materialize only the current batch, write it, clear it, repeat.
Why data engineers care
Batching balances throughput and memory: large enough for efficient writes, small enough to stay predictable.
Batch an iterator with itertools.isliceworked example
The representative pattern is list(islice(iterator, batch_size)): materialize a bounded window, not the whole stream.
Common mistake
Dropping the final partial batch. The last records in every file may disappear whenever the row count is not an exact multiple of batch size.
Better habit
Always yield the final non-empty batch.
Choose batch size based on downstream write behavior and memory.
Report batches_written and rows_written in the manifest.
Manifest habit
For batched writes, manifest counts should include rows_seen, rows_written, batches_written, and final_batch_size when useful.
Remember this
Batching is controlled materialization. Only hold the current chunk.
08 · Toolbelt
`itertools` For Pipeline Work
itertools gives you lazy operators. Treat them like streaming SQL operators over one-pass records.
⏱ 5 min · Topic 8 of 16
The itertools module provides iterator building blocks. For data engineering, the most useful ideas are slicing streams, chaining sources, grouping sorted records, and avoiding accidental materialization.
These tools are powerful because they preserve laziness. They are also dangerous if you forget one-pass behavior.
Use them when they make the pipeline simpler to read; avoid clever chains that hide business rules.
Core mental model
itertools gives you lazy operators. Treat them like streaming SQL operators over one-pass records.
Why data engineers care
Iterator tools let you express common streaming operations without custom buffering logic.
Using groupby on unsorted records and expecting SQL GROUP BY behavior. Only adjacent equal keys group together, so repeated keys later create separate groups.
Using tee on a huge stream while one branch lags far behind. Python may buffer many values for the slower branch.
Better habit
Sort before groupby when grouping by key.
Use islice for previews and batching.
Avoid tee unless you understand the buffering cost.
groupby warning
itertools.groupby groups consecutive records with the same key. It is not the same as SQL GROUP BY over an unordered table.
Remember this
itertools is a streaming toolbelt, not a reason to make the pipeline unreadable.
09 · Laziness
`groupby` And The One-Pass Traps
groupby groups runs, not keys. A group is a live view, not a list. Sort first, consume immediately.
⏱ 7 min · Topic 9 of 16
Lazy iteration buys memory, and it charges for it in a currency people forget: an iterator can only be walked once, and some of the standard library hands you views that expire.
`itertools.groupby` is the sharpest example. It groups **consecutive** equal keys, not all equal keys, so it produces the wrong groups unless the input is sorted by the same key. And each group it yields is a view into the shared iterator — advance to the next group and the previous one is empty.
The general rule underneath both: if you need the data twice, materialise it on purpose. Streaming is a choice with a cost, not a default that is always better.
Core mental model
groupby groups runs, not keys. A group is a live view, not a list. Sort first, consume immediately.
Why data engineers care
`groupby` on unsorted input produces a plausible result with too many groups, and stored groups produce empty lists. Neither raises, and both look like a data problem rather than a code one.
run
A stretch of adjacent items sharing a key. What `groupby` actually groups.
view
A lazy window onto a shared iterator. Valid only until the underlying iterator moves on.
materialise
Turn a lazy iterable into a list, deliberately, because you need it more than once.
groupby groups runs, not keysworked example
Python
fromitertoolsimportgroupbyevents=[("a",1),("a",2),("b",3),("a",4)]# Unsorted: 'a' appears twice, because the two runs are not adjacent.print([(k,len(list(g)))fork,gingroupby(events,key=lambdae:e[0])])# Sorted by the same key first: one group per key, as intended.ordered=sorted(events,key=lambdae:e[0])print([(k,list(g))fork,gingroupby(ordered,key=lambdae:e[0])])
The first result has three groups for two keys. That is not a bug in `groupby` — it groups adjacent runs, and adjacency is the caller’s job to arrange.
A group kept for later is an empty groupworked example
Python
fromitertoolsimportgroupbyevents=[("a",1),("a",2),("b",3)]# Storing the group objects and reading them AFTER the loop moved on.groups={k:gfork,gingroupby(events,key=lambdae:e[0])}print({k:list(g)fork,gingroups.items()})
Result · 1 row
output
{'a': [], 'b': []}
Every group is empty. Each one is a view into the same underlying iterator, and building the dict advanced past all of them. `list(g)` inside the loop is the fix.
An iterator is consumed onceworked example
Python
nums=iter([1,2,3])print(sum(nums),sum(nums))# the second pass sees nothing
Result · 1 row
output
6 0
The same trap without `groupby`. Any function that walks an iterable exhausts it if it was an iterator, and the second caller silently gets zero rows.
Grouping: which tool
Situation
Use
Why
Input already sorted by the key
`itertools.groupby`
Constant memory, one pass
Input unsorted, fits in memory
`defaultdict(list)`
One pass, no sort, no adjacency requirement
Input unsorted, does not fit
Sort externally, then `groupby`
The sort is the price of constant memory
You only need counts
`collections.Counter`
Cheaper than building the groups
Common mistake
Calling `groupby` without sorting by the same key first. A key appears once per run rather than once in total. The aggregate is wrong and the row count is plausible.
Storing the group objects to use after the loop. Every stored group reads as empty, because the shared iterator has already moved past it.
Passing a generator to two functions. The second gets nothing. `itertools.tee` or a list is the fix, and both cost memory — which is the trade you were avoiding.
Better habit
Sort by exactly the key you are about to group by — the same function, not an equivalent one.
`list(group)` inside the loop, always.
When two consumers need the same stream, decide explicitly between `tee`, a list, and reading twice.
Watch out
`itertools.tee` does not make iteration free. It buffers whatever one branch has consumed and the other has not, so two consumers at different speeds hold the difference in memory.
Interview note
"Why is the second pass empty?" is a standard output-prediction question. The answer is that an iterator holds its position, and generators do not restart.
Remember this
`groupby` groups adjacent runs and yields live views. Sort first, materialise inside the loop, and treat any second pass over a lazy source as a decision.
10 · State
Stateful Streaming Patterns
Streaming controls input buffering, not all memory. State size still matters.
⏱ 5 min · Topic 10 of 16
Not every streaming transform is stateless. Some need small state: counts by status, last event per user, a running total, or the current group for sorted input.
The key is bounded state. A running count by five statuses is safe. A dict of every user ever seen may become a memory problem if the population is huge.
When state grows with unique keys, ask whether the problem belongs in SQL, pandas, Spark, or a database-backed process instead.
Core mental model
Streaming controls input buffering, not all memory. State size still matters.
Why data engineers care
Stateful streaming bugs are subtle: the code is lazy, but the state dictionary may still grow until memory fails.
Assuming a generator is memory-safe even when it stores every seen key. The input stream is lazy, but the internal state still grows with the dataset.
Better habit
Name the state and estimate how it grows.
Prefer sorted-input group processing when possible.
Escalate to a warehouse or distributed engine when state is unbounded.
Senior signal
Say whether state is bounded. That one sentence separates a simple generator answer from a production-ready streaming answer.
Remember this
A streaming job is only memory-safe if both the input buffering and the internal state are bounded.
11 · Memory
Memory Safety And Backpressure Thinking
Every boundary either streams one item, buffers a bounded batch, or risks unbounded growth.
⏱ 5 min · Topic 11 of 16
Streaming pipelines have producers and consumers. A producer reads or parses records; a consumer validates, writes, uploads, or inserts them.
If the producer creates data faster than the consumer handles it, buffers grow. In pure file jobs, bounded batches usually solve this. In queue or network systems, backpressure becomes an explicit design concern.
Even in simple Python scripts, the habit is useful: never let an unbounded list become the hidden buffer between stages.
Core mental model
Every boundary either streams one item, buffers a bounded batch, or risks unbounded growth.
Why data engineers care
A memory incident often comes from an accidental buffer, not from the line that opened the file.
Write bounded batches instead of collecting all rowsworked example
Appending normalized rows to one big list before writing. The writer is batched, but memory still grows because the buffer is unbounded.
Better habit
Look for append calls inside long-running loops.
Bound every batch or buffer.
Emit row and batch counts in the manifest.
Memory review
During code review, scan for list accumulation and ask whether that list is bounded by batch size, source size, or nothing.
Remember this
Streaming is a memory contract. Every buffer needs a limit or a reason.
12 · Failure modes
Generator Failure Modes
Lazy code delays work. Delayed work means delayed errors, delayed side effects, and one-pass debugging.
⏱ 5 min · Topic 12 of 16
Generators make pipelines elegant, but they introduce their own failure modes. Errors may happen late, after the generator has been passed through several functions.
A generator can also hide resource lifetime issues. If a file is opened inside a generator, it should be consumed while the context manager is active.
Finally, one-pass streams make debugging trickier. Inspecting a stream can consume it unless you use a bounded preview carefully.
Core mental model
Lazy code delays work. Delayed work means delayed errors, delayed side effects, and one-pass debugging.
Why data engineers care
Streaming failures can appear far from the source line unless the pipeline preserves evidence and handles resource boundaries deliberately.
Preview a stream without losing the preview rowsworked example
preview rows chained back before the remaining stream
Common mistake
Logging list(stream) to debug and then passing stream onward. The downstream stage receives an exhausted iterator.
Returning a generator tied to a closed file handle. The caller gets an I/O error when it tries to consume the stream.
Better habit
Preserve source evidence inside yielded records.
Use bounded previews and chain rows back when needed.
Keep file iteration inside the context that owns the file handle.
Delayed error
Calling a generator function may not raise the error. The error often appears only when the caller starts iterating.
Remember this
Lazy pipelines need careful debugging and resource boundaries because work happens later.
13 · Tradeoffs
When Not To Stream
Choose between stream and list based on data size, access pattern, and required state.
⏱ 4 min · Topic 13 of 16
Streaming is not always the right answer. If the dataset is small and the logic needs repeated random access, a list may be clearer.
Some algorithms require sorting, joining on many keys, or comparing every record to every other record. Those may need materialization or a different engine.
Good engineering is choosing deliberately. Stream when memory or source shape demands it; materialize when it makes correctness clearer and the size is bounded.
Core mental model
Choose between stream and list based on data size, access pattern, and required state.
Why data engineers care
Overusing generators can make simple logic harder to debug without improving reliability.
Streaming decision table
Situation
Better default
Reason
Huge NDJSON file, row-level validation
stream
one record at a time is enough
Tiny config lookup table
list or dict
fits in memory and reused often
Need sort by timestamp
materialize or external sort
sorting needs the full comparison set
Warehouse-scale join
SQL or Spark
state is too large for one Python process
Common mistake
Using a generator chain for a ten-row lookup table. The code becomes harder to read with no practical memory benefit.
Better habit
Materialize small reference data deliberately.
Stream large fact-like data when processing is one-pass.
Move large joins, sorts, and windows to the right engine.
Tradeoff language
A strong answer says not just "I would stream it," but "I would stream the large event file and materialize the small country lookup as a dict."
Remember this
Streaming is a tool, not a religion. Use it where it improves correctness, memory, or throughput.
A strong streaming answer is concrete: it names the source, preserves evidence, yields records lazily, batches writes, and states what memory grows with.
Interviewers listen for whether you understand one-pass consumption and state size. Production reviewers look for unbounded buffers, missing line numbers, dropped final batches, and hidden tee buffers.
Use this chapter as a checklist before you ship any Python job that reads files or event streams.
Discussing yield without discussing memory, state, and failure paths. The solution sounds syntactic instead of production-ready.
Better habit
State the batch size and why it is bounded.
State how rejected rows are emitted.
State when you would stop streaming and use another engine.
What to say
I would stream the large file line by line, preserve line numbers, yield accepted and rejected records, write accepted rows in bounded batches, and report rows_seen, accepted, rejected, and batches_written.
Remember this
Professional streaming code is lazy, bounded, observable, and honest about when it should not be used.
15 · Practice
Practice Lab
Predict what will go wrong before you run it, then check whether you were right about which line caused it.
⏱ 6 min · Topic 15 of 16
Four exercises for this chapter, in the module workspace. Each one runs against graded cases, and each has a trap that is in the data rather than in the algorithm.
Four exercises where laziness is the design and its cost is the lesson.
Do them with the chapter closed. If one goes wrong, come back to the section it belongs to rather than re-reading the whole thing.
Core mental model
Predict what will go wrong before you run it, then check whether you were right about which line caused it.
Why data engineers care
Reading about a failure mode and meeting one are different memories. The second is the one that is still there under interview pressure.
Common mistake
Opening the reference solution before your own version runs. You learn what correct looks like without learning what wrong feels like, and the wrong version is the one you will write first under pressure.
Better habit
Write the contract of the function — what goes in, what comes out, what happens to a bad row — before the first line.
Run the empty-input case early. It is the one most solutions forget and the one every grader checks.
Study tip
Run each solution twice in the same process on two different inputs. Anything that behaves differently the second time is carrying state it should not.
Remember this
The chapter tells you what to watch for. These four are where you find out whether you would have.
