Iterators, Generators & Streaming Data

Openingloading the content…

Loadingone moment…

01 · Overview

Start here

⏱ 8 min · Topic 1 of 16

To iterate means to go through items one by one. You already do it with a for loop.

Anything a for loop can go through is called an iterable. A list is an iterable.

Here is the shop from the earlier chapters. One order is one row. In Python, one row is one dict.

Loop over the ordersworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

orders = [
    {"order_id": 101, "customer_id": 1, "status": "paid", "amount": 40.0},
    {"order_id": 102, "customer_id": 1, "status": "paid", "amount": 25.0},
    {"order_id": 103, "customer_id": 2, "status": "refunded", "amount": 60.0},
    {"order_id": 104, "customer_id": 3, "status": "paid", "amount": 15.5},
    {"order_id": 105, "customer_id": 3, "status": "pending", "amount": None},
    {"order_id": 106, "customer_id": 4, "status": "paid", "amount": 80.0},
]

for order in orders:
    print(order["order_id"], order["status"])

Result · 6 rows

output
101 paid
102 paid
103 refunded
104 paid
105 pending
106 paid

The loop gives you one order at a time. None means order 105 has no amount yet.

Real data is not a list in your code. It is a file with many lines. The same shop, as a small CSV file, looks like this:

The file orders.csvworked example

Python

order_id,customer_id,status,amount
101,1,paid,40.0
102,1,paid,25.0
103,2,refunded,60.0
104,3,paid,15.5
105,3,pending,
106,4,paid,80.0

CSV means comma-separated values. The first line is the header. It names the columns. Order 105 has an empty amount.

We use both forms in this chapter. The list shows each idea in a few lines. The file shows how the idea works on real input.

In every example below, orders is the list above and orders.csv is this file. The page shows the printed output, so you only need to read.

What you will learn

Why a loop that handles one row at a time beats loading all rows.
How to write a generator with yield.
Why a generator is used up after one pass.
How to chain small steps into a pipeline.
How to group rows into batches.

02 · Motivation

One row at a time

⏱ 8 min · Topic 2 of 16

Memory is the space where your program keeps its data while it runs. A list keeps all its rows in memory at once. This is fast, but the space is limited.

Think of a file with 10 GB (gigabytes) of orders. If you load every line into a list, the list needs room for all 10 GB. The program can run out of memory and crash.

First, load a file into a list. Look at the two printed values:

Load every line into a listworked example

Python

with open("orders.csv") as file:
    lines = file.readlines()

print(len(lines))
print(lines[1].strip())

Result · 2 rows

output
7
101,1,paid,40.0

The with line closes the file when the block ends. The call readlines() puts all 7 lines into one list: the header and six orders. The call strip() removes the new-line mark at the end of a line.

Now loop over the file instead. Python reads one line, you use it, and then Python reads the next line:

Read one line at a timeworked example

Python

count = 0
with open("orders.csv") as file:
    for line in file:
        count += 1

print(count)

Result · 1 row

output
7

The answer is the same. But the program only held one line at any moment.

Rows that you read one after another are called a stream.

A list against one row at a time

	List of all rows	One row at a time
Rows in memory	All of them	One
Work starts	After everything is loaded	At the first row
Go back to row 1	Yes, any time	No

The last row of the table is the cost. You cannot go back. The rest of the chapter shows what to do.

Interview question

“Why not load a 10 GB file into a list?” The list needs all 10 GB of memory. Read one line at a time instead.

03 · Core idea

Iterables and iterators

⏱ 6 min · Topic 3 of 16

A for loop does two things. First it asks the iterable for an iterator. Then it asks the iterator for the next item, again and again.

An iterator is an object that remembers where you are. iter() makes one. next() gives the next item. Watch the position move:

iter() and next()worked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

it = iter(orders)

print(next(it)["order_id"])
print(next(it)["order_id"])

print("rest:")
for order in it:
    print(order["order_id"])

Result · 7 rows

output
101
102
rest:
103
104
105
106

next(it) gives one order (a dict). ["order_id"] picks its id. Two next() calls took orders 101 and 102. The loop then continued from 103.

What happens when no items are left? next() raises an error called StopIteration. A for loop catches it for you and stops the loop.

Nothing left: StopIterationworked example

Python

numbers = iter([10, 20])

print(next(numbers))
print(next(numbers))
print(next(numbers))

Result · 3 rows

output
10
20
StopIteration

The third next() has no item, so Python stops with the error StopIteration.

A list is an iterable, but it is not an iterator. A list does not remember a position, so you can loop over it many times. An iterator remembers its position, so it only goes forward.

A file is an iterator

The file in the last section is an iterator too. So a for loop over it reads one line at a time. You cannot restart it without opening the file again.

04 · Generators

Write a generator with yield

⏱ 8 min · Topic 4 of 16

A generator is a function that gives its results one at a time. You write it with yield instead of return.

Calling a generator function does not run its code. It returns an iterator.

This is called lazy: the work waits until someone asks for the next item.

The code runs only when you ask for the next item. At each yield, the function pauses and remembers where it was. Watch the prints:

A generator pauses at yieldworked example

Python

def count_up():
    print("start")
    yield 1
    print("middle")
    yield 2
    print("end")

numbers = count_up()
print("made it")
print(next(numbers))
print(next(numbers))

Result · 5 rows

output
made it
start
1
middle
2

count_up() printed nothing. The first next() ran the code up to yield 1. The second next() resumed and ran up to yield 2. "end" never printed, because nobody asked for a third item.

Now a useful generator. It goes through the orders and gives only the paid ones:

A generator for paid ordersworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

def paid_orders(orders):
    for order in orders:
        if order["status"] == "paid":
            yield order

for order in paid_orders(orders):
    print(order["order_id"], order["amount"])

Result · 4 rows

output
101 40.0
102 25.0
104 15.5
106 80.0

Orders 103 (refunded) and 105 (pending) never came out. The function never built a list of paid orders.

Because the code runs late, a bug inside a generator shows up late too. It appears when you start the loop, not when you call the function.

Watch out: return stops the generator

If you write return order instead of yield order, the function is no longer a generator. It gives back one dict and stops at the first paid order. The for loop then fails with an error, because it loops over the dict keys.

Interview question

“What is the difference between a list and a generator?” A list holds all its items in memory. A generator makes one item when you ask for it, and you can read it only once.

05 · Short form

Comprehensions and generator expressions

⏱ 6 min · Topic 5 of 16

A comprehension builds a collection in one line. A list comprehension builds a list, with square brackets. The records chapter showed the dict and set forms.

A list comprehensionworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

paid_ids = [
    order["order_id"] for order in orders if order["status"] == "paid"
]
print(paid_ids)

Result · 1 row

output
[101, 102, 104, 106]

Read it as: take order_id, for each order, if the status is paid. The list holds all four ids.

Round brackets give you a generator expression. It makes a generator, not a list.

A generator expressionworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

paid_amounts = (
    order["amount"] for order in orders if order["status"] == "paid"
)
print(sum(paid_amounts))

Result · 1 row

output
160.5

sum() asked for the amounts one at a time: 40.0 + 25.0 + 15.5 + 80.0. No list of amounts was built.

Square or round brackets

Form	Makes	Use it when
[x for ...]	A list, right now	You need the items more than once
(x for ...)	A generator, later	You loop once, for example inside sum()

What if you leave out the paid filter? Order 105 has a None amount:

The None amount breaks sum()worked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

print(sum(order["amount"] for order in orders))

Result · 1 row

output
TypeError: unsupported operand type(s) for +: 'float' and 'NoneType'

Python cannot add None to a number. The paid filter kept order 105 out of the sum before.

06 · Interview topic

A generator runs out

⏱ 8 min · Topic 6 of 16

A generator gives each item once. After the last item, it is used up, and it does not restart.

Here we use the same generator expression twice:

The second pass is emptyworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

amounts = (
    order["amount"] for order in orders if order["status"] == "paid"
)

print(sum(amounts))
print(sum(amounts))

Result · 2 rows

output
160.5
0

The first sum() used up all four amounts. The second sum() found nothing, and the sum of nothing is 0.

There is no error. You just get a wrong answer. With a list, the same code gives the right answer both times:

A list can be read againworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

amounts = [
    order["amount"] for order in orders if order["status"] == "paid"
]

print(sum(amounts))
print(sum(amounts))

Result · 2 rows

output
160.5
160.5

A quick count causes the same bug. Counting the rows first uses them up:

Counting uses up the rowsworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

paid = paid_orders(orders)

print("count:", len(list(paid)))
for order in paid:
    print(order["order_id"])
print("after the loop")

Result · 2 rows

output
count: 4
after the loop

The loop printed no order ids. list(paid) already went through all four orders.

You have three fixes:

Call the generator function again to get a fresh generator.
Turn the generator into a list with list(), when the data is small enough for memory.
Do everything in one pass: count and process in the same loop.

Interview question

“Why is the second pass empty?” An iterator remembers its position. A generator is an iterator, so it cannot go back. Say the fix too: make a new generator, or use a list.

07 · Pipelines

How rows travel

⏱ 5 min · Topic 7 of 16

A pipeline is a chain of small generators. Each step takes rows from the step before and yields rows to the step after.

Here is a tiny pipeline with two steps: read and label. The read step uses orders[:2], which means the first two orders only.

Each row goes through both steps before the next row starts. Follow the prints:

One row travels through all stepsworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

def read(orders):
    for order in orders[:2]:
        print("read", order["order_id"])
        yield order

def label(orders):
    for order in orders:
        print("label", order["order_id"])
        yield order["order_id"], order["status"]

for item in label(read(orders)):
    print("got", item)

Result · 6 rows

output
read 101
label 101
got (101, 'paid')
read 102
label 102
got (102, 'paid')

Order 101 went through read, label and the loop. Only then did read start on order 102. No step held both rows.

Each step accepts any iterable. So you can test a step with a short list, with no file at all.

08 · Pipelines

Build the file pipeline

⏱ 8 min · Topic 8 of 16

Now the real pipeline for the shop: read, clean, filter, then output. The rows flow through these five parts:

orders.csv: the file.
read_rows: splits each line into a dict of text.
clean_rows: turns the text into numbers.
only_paid: keeps the paid rows.
A for loop: prints each row. This is the output step.

Step 1 reads each line and splits it at the commas. Every value is still text:

Step 1: readworked example

Python

def read_rows(path):
    with open(path) as file:
        next(file)  # skip the header line
        for line in file:
            order_id, customer_id, status, amount = line.strip().split(",")
            yield {
                "order_id": order_id,
                "customer_id": customer_id,
                "status": status,
                "amount": amount,
            }

print(next(read_rows("orders.csv")))

Result · 1 row

output
{'order_id': '101', 'customer_id': '1', 'status': 'paid', 'amount': '40.0'}

A # starts a comment. Python ignores it. The four names on the left take the four values from the split. The quotes around 101 and 40.0 show they are text.

Step 2 cleans each row. It turns the ids into whole numbers with int(). It turns the amount into a decimal number with float().

Step 3 keeps only the paid rows. Both steps are generators:

Steps 2 and 3: clean and filterworked example

Python

def clean_rows(rows):
    for row in rows:
        yield {
            "order_id": int(row["order_id"]),
            "customer_id": int(row["customer_id"]),
            "status": row["status"],
            "amount": float(row["amount"]) if row["amount"] else None,
        }

def only_paid(rows):
    for row in rows:
        if row["status"] == "paid":
            yield row

Read the amount line as: if the amount text is not empty, use float(...). Otherwise use None. An empty text counts as false, so order 105 gets None. Calling float("") would fail.

Last, connect the steps and loop over the result. The loop is the output step:

The whole pipelineworked example

Python

rows = read_rows("orders.csv")
cleaned = clean_rows(rows)
paid = only_paid(cleaned)

for row in paid:
    print(row)

Result · 4 rows

output
{'order_id': 101, 'customer_id': 1, 'status': 'paid', 'amount': 40.0}
{'order_id': 102, 'customer_id': 1, 'status': 'paid', 'amount': 25.0}
{'order_id': 104, 'customer_id': 3, 'status': 'paid', 'amount': 15.5}
{'order_id': 106, 'customer_id': 4, 'status': 'paid', 'amount': 80.0}

Nothing ran until the for loop started. Then each line of the file went through all three steps, one at a time.

Interview question

“Design a streaming pipeline in plain Python.” Say: one small generator per step. Each takes an iterable and yields rows. Nothing is stored. A final loop pulls the rows through.

09 · Batches

Group rows into batches

⏱ 6 min · Topic 9 of 16

Databases and files often work better when you write many rows at once. A batch is a group of rows of a fixed size. Only one batch is in memory at a time.

We need itertools.islice. It takes the next few items from an iterator.

When fewer items are left, it takes what is left. When none are left, it gives nothing.

islice takes the next few itemsworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

from itertools import islice

it = iter(orders)
first = list(islice(it, 4))
second = list(islice(it, 4))
third = list(islice(it, 4))

print(len(first), len(second), len(third))

Result · 1 row

output
4 2 0

Six orders: the first call took 4, the second took the last 2, and the third found none.

Now put islice in a generator. It yields one batch each time:

A batches generatorworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

from itertools import islice

def batches(items, size):
    iterator = iter(items)
    while True:
        batch = list(islice(iterator, size))
        if not batch:
            return
        yield batch

for batch in batches(orders, 4):
    print([order["order_id"] for order in batch])

Result · 2 rows

output
[101, 102, 103, 104]
[105, 106]

The loop while True repeats until return runs. An empty list counts as false, so not batch means no rows were left. Then return ends the generator. The last batch has only two orders, and it is still yielded. If the code kept only full batches, orders 105 and 106 would be lost without any error.

Python 3.12

Python 3.12 has itertools.batched(orders, 4). It does the same job, but each batch is a tuple, not a list.

10 · Memory

Keep memory small

⏱ 5 min · Topic 10 of 16

A lazy pipeline can still fill memory with a hidden list. Look for append inside a long loop:

A hidden list holds every rowworked example

Python

rows = read_rows("orders.csv")
cleaned = clean_rows(rows)
paid = only_paid(cleaned)

results = []
for row in paid:
    results.append(row)

print(len(results))

Result · 1 row

output
4

The pipeline was lazy, but results is not. With a 10 GB file, results holds every paid row.

Instead, write each batch, then drop it. This code uses the steps and the batches generator from the earlier sections. It also counts what it wrote:

Write one batch at a timeworked example

Python

paid = only_paid(clean_rows(read_rows("orders.csv")))

rows_written = 0
batches_written = 0

for batch in batches(paid, 2):
    print("write", [row["order_id"] for row in batch])
    rows_written += len(batch)
    batches_written += 1

print({"rows_written": rows_written, "batches_written": batches_written})

Result · 3 rows

output
write [101, 102]
write [104, 106]
{'rows_written': 4, 'batches_written': 2}

The print stands in for a real write to a database. At most 2 rows were in memory. The counts show nothing was lost.

When you check code, look at every list that grows. Ask what limits its size. The answer should be a batch size, not the size of the input.

11 · Trade-offs

When a list is better

⏱ 5 min · Topic 11 of 16

Streaming is not always the best choice. Some jobs need all the rows at once. Sorting is one: the smallest row could be the last one.

The next code uses three new pieces:

key tells sorted() what to compare.
lambda o: o["amount"] is a tiny function. It takes an order and returns its amount.
reverse=True puts the biggest first.

Sorting builds a listworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

biggest_first = sorted(
    paid_orders(orders),
    key=lambda o: o["amount"],
    reverse=True,
)

print([order["order_id"] for order in biggest_first])

Result · 1 row

output
[106, 101, 102, 104]

sorted() pulled all four paid orders from the generator and returned a list. Amounts 80.0, 40.0, 25.0, 15.5 give ids 106, 101, 102, 104.

Stream or list

Situation	Use	Why
A huge file, checked row by row	Stream	One row at a time is enough
A small lookup, such as 4 customers	Dict or list	It fits in memory and you use it many times
You need to sort	List	Sorting needs every row
You need to loop twice	List	A generator is empty on the second pass

In an interview, say you use both: stream the large order file and keep a small customer lookup (a dict) in memory. For very large joins or sorts, use SQL or Spark instead of one Python process.

12 · Optional

Preview and join streams

⏱ 5 min · Topic 12 of 16

Optional: two more itertools tools help in pipelines. islice lets you look at the first rows. chain joins several streams into one.

Looking at the first rows has a catch. The rows you take are gone from the stream. Put them back in front with chain:

Preview without losing rowsworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

from itertools import chain, islice

stream = iter(orders)
preview = list(islice(stream, 2))
whole = chain(preview, stream)

print([order["order_id"] for order in preview])
print([order["order_id"] for order in whole])

Result · 2 rows

output
[101, 102]
[101, 102, 103, 104, 105, 106]

chain gave the two preview rows first, then the rest of the stream. Without chain, the second line would start at 103.

chain also joins two sources. Imagine one file for each day.

Day 1 has orders 101 to 103. Day 2 has the rest:

Two sources as one streamworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

from itertools import chain

day_1 = orders[:3]
day_2 = orders[3:]

print([order["order_id"] for order in chain(day_1, day_2)])

Result · 1 row

output
[101, 102, 103, 104, 105, 106]

orders[:3] is the first three orders. orders[3:] is all the others. chain reads day_1, then day_2. It never copies them into a new list.

13 · Optional

groupby: runs, not keys

⏱ 6 min · Topic 13 of 16

Optional: read this when you first meet itertools.groupby. It groups rows that have the same key. But it only groups rows that sit next to each other.

A key is the value you group by. Here the key is the status, the first item of each pair.

The tiny function lambda e: e[0] is like the one in the sorting section. It returns the first item.

Here is a small new list of events. Each event is a pair: the status and the order id:

groupby groups runs, not keysworked example

Python

from itertools import groupby

events = [("paid", 101), ("paid", 102), ("refunded", 103), ("paid", 104)]

# Unsorted: "paid" appears twice, because the two runs are not adjacent.
print([(k, len(list(g))) for k, g in groupby(events, key=lambda e: e[0])])

# Sorted by the same key first: one group per key.
ordered = sorted(events, key=lambda e: e[0])
print([(k, list(g)) for k, g in groupby(ordered, key=lambda e: e[0])])

Result · 2 rows

output
[('paid', 2), ('refunded', 1), ('paid', 1)]
[('paid', [('paid', 101), ('paid', 102), ('paid', 104)]), ('refunded', [('refunded', 103)])]

The first result has three groups for two keys. That is not a bug in groupby. It only groups rows that sit next to each other, so you must sort the rows first.

There is a second trap. All groups share one stream. When the loop moves on, the old group is used up.

A group kept for later is emptyworked example

Python

from itertools import groupby

events = [("paid", 101), ("paid", 102), ("refunded", 103)]

groups = {k: g for k, g in groupby(events, key=lambda e: e[0])}
print({k: list(g) for k, g in groups.items()})

Result · 1 row

output
{'paid': [], 'refunded': []}

Every group is empty. Building the dict moved past all of them. Use list(g) inside the loop.

Which tool to group with

Situation	Use
Rows already sorted by the key	groupby
Rows not sorted, and they fit in memory	A dict where each key has a list
You only need counts	A dict of counts

14 · Optional

Keep a running count

⏱ 4 min · Topic 14 of 16

Optional: some steps need to remember something. State is the data a step keeps between rows. Here the state is a count of orders per status.

A running count per statusworked example

Python

Input data

orders6 rows

order_id	customer_id	status	amount
101	1	paid	40
102	1	paid	25
103	2	refunded	60
104	3	paid	15.5
105	3	pending	NULL
106	4	paid	80

def with_running_count(orders):
    counts = {}
    for order in orders:
        status = order["status"]
        counts[status] = counts.get(status, 0) + 1
        yield order["order_id"], status, counts[status]

for row in with_running_count(orders):
    print(row)

Result · 6 rows

output
(101, 'paid', 1)
(102, 'paid', 2)
(103, 'refunded', 1)
(104, 'paid', 3)
(105, 'pending', 1)
(106, 'paid', 4)

counts.get(status, 0) gives the count so far, or 0 if the status is new. Order 104 is the third paid order, so its count is 3. The dict counts holds only three keys.

This state is small and stays small. There are only a few statuses, even if the file has billions of rows.

Watch out: state can grow

A dict with one key per customer grows with every new customer. The stream is lazy, but memory still fills up. Ask what the state grows with.

Iterators, Generators & Streaming Data

Start here

What you will learn

One row at a time

Iterables and iterators

Write a generator with yield

Comprehensions and generator expressions

A generator runs out

How rows travel

Build the file pipeline

Group rows into batches

Keep memory small

When a list is better

Preview and join streams

groupby: runs, not keys

Keep a running count

Practice lab

Next Chapter

Iterators, Generators & Streaming Data

Start here

What you will learn

One row at a time

Iterables and iterators

Write a generator with yield

Comprehensions and generator expressions

A generator runs out

How rows travel

Build the file pipeline

Group rows into batches

Keep memory small

When a list is better

Preview and join streams

groupby: runs, not keys

Keep a running count

Practice lab

Next Chapter