Records, Collections & Keyed Data

In chapter 1, a record became one dict, and a dataset became a list of records. This chapter shows the tools that hold and reshape those records.

Here is the shop from the SQL chapters as Python data. Each order is one dict. The three lists are the data for every example in this chapter:

orders = [
    {"order_id": 101, "customer_id": 1, "status": "paid",
     "amount": 40.0, "order_date": "2026-01-03"},
    {"order_id": 102, "customer_id": 1, "status": "paid",
     "amount": 25.0, "order_date": "2026-01-10"},
    {"order_id": 103, "customer_id": 2, "status": "refunded",
     "amount": 60.0, "order_date": "2026-01-11"},
    {"order_id": 104, "customer_id": 3, "status": "paid",
     "amount": 15.5, "order_date": "2026-01-15"},
    {"order_id": 105, "customer_id": 3, "status": "pending",
     "amount": None, "order_date": "2026-01-20"},
    {"order_id": 106, "customer_id": 4, "status": "paid",
     "amount": 80.0, "order_date": "2026-01-21"},
]
customers = [
    {"customer_id": 1, "name": "Asha", "country": "IN"},
    {"customer_id": 2, "name": "Ben", "country": "GB"},
    {"customer_id": 3, "name": "Carla", "country": "US"},
    {"customer_id": 4, "name": "Dev", "country": None},
    {"customer_id": 5, "name": "Omar", "country": "IN"},
]
order_items = [
    {"order_id": 101, "product": "Pen", "quantity": 2},
    {"order_id": 101, "product": "Notebook", "quantity": 1},
    {"order_id": 102, "product": "Bag", "quantity": 1},
    {"order_id": 103, "product": "Lamp", "quantity": 1},
    {"order_id": 104, "product": "Pen", "quantity": 3},
    {"order_id": 106, "product": "Chair", "quantity": 1},
    {"order_id": 106, "product": "Lamp", "quantity": 1},
]

print(len(orders))
print(orders[0])

What you will learn

A collection is one value that holds many values. Python has four main collections:

• list: items in order. You can change it.
• tuple: items in order. You cannot change it.
• dict: pairs of a key and a value.
• set: different values, with no repeats.

Four tools in this chapter match SQL ideas you know:

You also learn why a dict lookup is fast, and how to use two fields together as one key.

A dict stores pairs. Each pair has a key and a value. In a record, the key is the field name.

You read a value with its key in square brackets. This reads three fields of order 103, then lists its keys:

order = orders[2]
print(order["order_id"], order["status"], order["amount"])
print(list(order))

list(a_dict) gives the keys as a list. sorted(a_dict) gives the keys in sorted order.

If the key is not in the dict, Python stops with an error. Order 103 has no discount field:

order = orders[2]
print(order["discount"])

You know .get from chapter 1. It gives None, or your default, when the key is missing.

Use square brackets for a field every record has. Use .get for a field that may be missing.

A default only helps when the key is missing. Order 105 has the key amount, and its value is None. Compare the two reads:

order = orders[4]
print(order.get("amount", 0))

amount = order["amount"]
if amount is None:
    amount = 0
print(amount)

A list keeps items in order, and you can add, remove or change items. The shop keeps orders in a list.

You can loop over a list. This loop collects the order_id of each paid order:

paid_ids = []
for o in orders:
    if o["status"] == "paid":
        paid_ids.append(o["order_id"])
print(paid_ids)

A list comprehension is the same loop in one line. Read it as: take o["order_id"], for each o in orders, if the status is paid:

paid_ids = [o["order_id"] for o in orders if o["status"] == "paid"]
print(paid_ids)

A tuple also keeps items in order, but you cannot change it after you make it. Round brackets make a tuple. Unpacking gives each item its own name:

line = (101, "Pen")
order_id, product = line
print(order_id, product)

If you try to change a tuple, Python gives an error:

line = (101, "Pen")
line[0] = 102

Tuples have one more use as dict keys. It is shown in the section on dedupe and composite keys.

A join attaches fields from one table to another. In Python, you do it with a lookup dict.

You build it once. The key is customer_id, and the value is the whole customer record.

This builds a lookup for customers, then reads customer 2. The curly-bracket line is a dict comprehension. It is like a list comprehension, but it makes key: value pairs:

customer_by_id = {c["customer_id"]: c for c in customers}
print(customer_by_id[2])

Now loop over the orders. For each order, look up its customer by customer_id, and build a new record with the name:

joined = []
for order in orders:
    customer = customer_by_id[order["customer_id"]]
    joined.append({"order_id": order["order_id"], "name": customer["name"]})

for row in joined:
    print(row)

When the key is missing

Imagine a new order for customer 9, who is not in customers. Square brackets would raise KeyError: 9, as in the dicts section.

The method .get gives None instead. Your code can then choose a default value:

customer = customer_by_id.get(9)
if customer is None:
    name = "unknown"
else:
    name = customer["name"]
print(name)

This is the same choice as in SQL. Square brackets need a match, like an inner join. The method .get is like a LEFT JOIN: the order stays, with no customer.

When a key repeats

A dict holds one value for each key. If two customers share an id, the later one replaces the earlier one. Python gives no error:

rows = [
    {"customer_id": 1, "name": "Asha"},
    {"customer_id": 1, "name": "Asha B"},
]
print({r["customer_id"]: r["name"] for r in rows})

A lookup dict is fast because Python turns the key into a number. The number says where to look.

A list has no such shortcut. It must check the items one by one.

Here is the join from the last section with a loop inside a loop. The counter counts each comparison of two ids. checks += 1 means: add 1 to checks.

break stops the inner loop as soon as the customer is found:

checks = 0
for order in orders:
    for customer in customers:
        checks += 1
        if customer["customer_id"] == order["customer_id"]:
            break
print(checks)

With the lookup dict, the same join needs 5 steps to build the dict and 6 lookups. That is 11 steps.

Now the sizes grow. In the table, n is the number of customers and m is the number of orders:

The loop stops early when it finds the customer, so the real count (14) is below the worst case (30).

Computer scientists describe this growth with O(...). O(n) means the work grows in step with n: ten times more rows, ten times more work.

Scanning a list for one item is O(n). A dict lookup is O(1) on average: the work stays about the same, however many rows the dict holds.

A nested loop is O(n × m), the two sizes multiplied. That is the third column of the table.

Grouping puts records with the same key into one bucket. Here, a bucket is a list for that key. It is GROUP BY in Python.

A defaultdict is a dict that makes a missing key for you. The form defaultdict(list) makes an empty list at the first use of a key.

So you can append at once. A plain dict would raise a KeyError. This groups the order ids by customer:

from collections import defaultdict

orders_by_customer = defaultdict(list)
for order in orders:
    customer_id = order["customer_id"]
    orders_by_customer[customer_id].append(order["order_id"])

print(dict(orders_by_customer))

Omar is customer 5. He has no orders, so key 5 is not in the dict.

This uses orders_by_customer from the last example. Use .get with an empty list as the default:

print(orders_by_customer.get(5, []))
print(5 in orders_by_customer)

Group first. Then turn each list into one number.

.items() gives each key and its value, one pair at a time. sum adds up a list of numbers.

This adds up the amounts for each customer. It skips order 105, because its amount is None:

amounts = defaultdict(list)
for order in orders:
    if order["amount"] is not None:
        amounts[order["customer_id"]].append(order["amount"])

totals = {cid: sum(values) for cid, values in amounts.items()}
print(totals)

Reading a missing key on a defaultdict is different from .get. It makes the key. This uses orders_by_customer from the first example:

print(orders_by_customer[5])
print(sorted(orders_by_customer))

Count with Counter

Counting how often each value appears is so common that Python has a tool for it. Counter is a dict that counts. This counts the orders for each status:

from collections import Counter

counts = Counter([o["status"] for o in orders])
print(counts)
print(counts["paid"], counts["cancelled"])
print(counts.most_common(2))

To add a number instead of 1, use += on each key. This counts the units sold for each product:

units = Counter()
for item in order_items:
    units[item["product"]] += item["quantity"]

print(units.most_common(2))

A set holds different values. It drops repeats, and it does not promise any order.

Curly brackets with a for build a set. This is a set comprehension, like a list comprehension. sorted(...) puts the values in order, as a list. This finds the customers who placed an order:

ordered_ids = {o["customer_id"] for o in orders}
print(sorted(ordered_ids))
print(len(orders), len(ordered_ids))

Sets can also subtract. The difference a - b gives the values in a that are not in b. Which customers never ordered?

all_ids = {c["customer_id"] for c in customers}
ordered_ids = {o["customer_id"] for o in orders}

print(all_ids - ordered_ids)
print(ordered_ids - all_ids)
print(4 in ordered_ids, 5 in ordered_ids)

This is the Python version of an anti-join: find the keys that have no match on the other side. The last line checks one value with in.

To dedupe means to remove repeats. You must decide two things: what makes two rows the same, and which one to keep.

In the shop, customers 1 and 3 each have two orders. Use customer_id as the key.

This keeps the first order of each customer. A set remembers the keys you have already seen:

seen = set()
first_orders = []
for order in orders:
    customer_id = order["customer_id"]
    if customer_id not in seen:
        seen.add(customer_id)
        first_orders.append(order["order_id"])

print(first_orders)

To keep the last row instead, write the rows into a dict. A later row replaces an earlier row with the same key:

latest = {o["customer_id"]: o["order_id"] for o in orders}
print(latest)

For plain values, there is a short way. A dict keeps its keys in order, and a key cannot repeat:

customer_ids = [o["customer_id"] for o in orders]
print(customer_ids)
print(list(dict.fromkeys(customer_ids)))

Keys made of two fields

Sometimes one field is not enough to name a row. In order_items, order_id repeats: order 101 has two lines. product repeats too: Pen is in orders 101 and 104.

Together, (order_id, product) is unique. A key made of more than one field is a composite key.

In Python, a composite key is a tuple. But a dict key must be hashable.

Hashable means Python can turn the value into one fixed number. Numbers, text and tuples of them are hashable. Lists and dicts are not, because they can change.

This builds a quantity lookup, and compares it with a lookup by product only:

by_line = {(i["order_id"], i["product"]): i["quantity"] for i in order_items}
by_product = {i["product"]: i["quantity"] for i in order_items}

print(len(order_items), len(by_line), len(by_product))
print(by_line[(106, "Lamp")])
print(by_product["Pen"])

Look at the last line. by_product["Pen"] is 3, the quantity of order 104. The 2 pens of order 101 were replaced, and Python said nothing.

A list as a key is an error:

quantity = {[101, "Pen"]: 2}

You can use a tuple in a set too. This checks if one order contains one product:

lines = {(i["order_id"], i["product"]) for i in order_items}
print((101, "Pen") in lines, (101, "Bag") in lines)

A record can hold a list of other records. An API often sends an order with its items inside it. This is a nested record.

This groups the items by order, then puts the items of order 101 inside the order:

from collections import defaultdict

items_by_order = defaultdict(list)
for item in order_items:
    items_by_order[item["order_id"]].append(
        {"product": item["product"], "quantity": item["quantity"]}
    )

order = orders[0]
nested = {"order_id": order["order_id"], "items": items_by_order[order["order_id"]]}
print(nested)

To read inside a nested record, chain .get calls. A missing level gives None, not an error:

print(nested.get("customer", {}).get("name"))

A table needs flat rows. To flatten, make one row for each item. In the code, **item copies the fields of item into the new dict.

For the first item, {"order_id": order_id, **item} is {"order_id": 101, "product": "Pen", "quantity": 2}:

flat = []
for order_id, items in items_by_order.items():
    for item in items:
        flat.append({"order_id": order_id, **item})

print(len(orders), len(flat))
print(flat[0])

Order 101 had one record and now has two rows. When one record becomes many rows, the row count grows. This is called fan-out.

Order 105 has no items, so it disappeared. Compare row counts before and after every join.

Each exercise opens in the Python workspace with its own data. Write the code, run it, and check the result against the tests.

They follow the order of this chapter. Each one has a trap in the data, not in the loop.