The Python Data Model: Names, Objects & Mutability

Some Python bugs give no error. The code runs, the numbers look fine, and the numbers are wrong. Most of them come from one idea: how a name relates to the data it holds.

You already know these words from earlier chapters:

• A variable is a name that holds a value.
• A list keeps values in order. A tuple is a list that cannot change.
• A dict keeps values under keys. A set keeps each value once.
• A function is a block of code with a name. You call it to run it.

The examples use the same shop. The list `orders` holds six dicts, one for each order.

What you will learn

1. A name points at an object. Two names can point at one object.
2. Which objects can change (mutable) and which cannot (immutable).
3. Why a function can change the data you gave it.
4. The default argument bug that carries rows over from one call to the next.
5. The difference between a shallow copy and a deep copy.
6. The difference between `==` and `is`.
7. Why a dict key must not change.
8. (Optional) What a small function remembers about names outside it.

An object is the thing Python keeps in memory. A dict is an object. A list is an object.

A name is a label that points at an object. The sign `=` makes a name point at an object. It does not make a copy.

Read these lines. The second line gives the dict a second name:

order = {"order_id": 101, "status": "paid", "amount": 40.0}
same = order
same["status"] = "refunded"

print(order)
print(order is same)

There is only one dict. The test `order is same` asks if both names point at it. The answer is True.

This table shows what each name points at after each line:

Moving a name is different

A line like `same = something` moves the name. It does not touch the old object. Look at what happens to the first dict:

order = {"order_id": 101, "status": "paid"}
same = order
same = {"order_id": 999, "status": "new"}

print(order)
print(same)

This table shows where the names point after the last line:

A mutable object can change after you make it. An immutable object cannot change.

This table shows which common types are which:

Watch what a string does and what a list does:

status = "paid"
status.upper()          # makes a new string "PAID" that we do not keep
print(status)

amounts = [40.0, 25.0]
amounts.append(60.0)    # changes the list itself
print(amounts)

A tuple gives an error when you try to change it. The row below is a tuple:

row = (101, "paid")
try:
    row[1] = "refunded"
except TypeError as error:
    print(error)

A mutable object shared by two names can change under both. An immutable one cannot.

When you call a function with a dict, the function gets another name for your dict. It does not get a copy. If the function changes the dict, your dict changes.

This function adds a key `checked` to an order and returns the order:

def mark_checked(order):
    order["checked"] = True
    return order

order = {"order_id": 101, "status": "paid", "amount": 40.0}
checked = mark_checked(order)

print(order)
print(checked is order)

A change a function makes outside its result is a side effect. The call `mark_checked(order)` looks like it only returns a value. It also changed your `order`.

The fix is to build a new dict. The form `{**order, "checked": True}` copies every key of `order` into a new dict, then adds the new key:

def with_checked(order):
    return {**order, "checked": True}

order = {"order_id": 101, "status": "paid", "amount": 40.0}
checked = with_checked(order)

print(order)
print(checked)

Choose one way for each function. Either it changes its input and returns nothing, or it returns a new object and leaves its input alone. A function that does both can hide the change.

A default argument is the value Python uses when the caller gives none. Python creates that value once, when the `def` line runs. It does not create it again on each call.

So a list as a default is one list, shared by every call. This function collects the rejected orders from a batch. A rejected order is one with no amount:

def find_rejected(batch, rejected=[]):
    for order in batch:
        if order["amount"] is None:
            rejected.append(order)
    return rejected

batch_1 = [{"order_id": 104, "amount": 15.5}, {"order_id": 105, "amount": None}]
batch_2 = [{"order_id": 106, "amount": 80.0}]

print(find_rejected(batch_1))
print(find_rejected(batch_2))

The fix is to use None as the default, and build the list inside the function. None is immutable, so sharing it is safe. Run it on the same two batches:

def find_rejected(batch, rejected=None):
    if rejected is None:
        rejected = []
    for order in batch:
        if order["amount"] is None:
            rejected.append(order)
    return rejected

print(find_rejected(batch_1))
print(find_rejected(batch_2))

Use "if rejected is None", not "if not rejected".

An empty list from the caller counts as false, so "not rejected" is True. The function would build a new list. The caller's list would stay empty, and the caller would not see the rejected orders.

A copy is a new object with the same content. There are two kinds.

A shallow copy makes a new outer list, but the dicts inside stay shared. A deep copy also makes new dicts inside. This table shows the common ways to copy:

Start again with a new list `orders` that has two dicts. Try the shallow copy first:

orders = [
    {"order_id": 101, "amount": 40.0},
    {"order_id": 102, "amount": 25.0},
]

backup = list(orders)
backup.append({"order_id": 103, "amount": 60.0})
print(len(orders), len(backup))

backup[0]["amount"] = 0.0
print(orders[0])

This table shows who points at what after both copies. The deep copy is in the next example:

A deep copy makes new dicts too. Continue with the same `orders`:

import copy

safe = copy.deepcopy(orders)
safe[1]["amount"] = 99.0

print(orders[1])
print(safe[1])

A shallow copy is enough when the inside items are only numbers and text. Use a deep copy when a list or dict is inside.

A deep copy is slow on big data. Copy once at the start, not inside a loop.

The test `==` asks: do two objects have the same value? The test `is` asks: are they the same object?

Two dicts built separately can be equal and still be two objects:

a = {"order_id": 101, "status": "paid"}
b = {"order_id": 101, "status": "paid"}
c = a

print(a == b)
print(a is b)
print(a is c)

Use `==` to compare values. Use `is` only for `None`, because there is only one `None` in Python. This loop finds the orders with no amount:

for order in orders:
    if order["amount"] is None:
        print(order["order_id"])

A dict finds a key by a number that Python works out from the key. That number is the hash. A set uses the same method.

If a key could change, its number would change, and Python could not find it again. So only immutable objects can be keys. We call them hashable.

An object that cannot be hashed gives an error.

A tuple is a good key. This code counts the different pairs of customer and status in the six orders:

pairs = set()
for order in orders:
    pairs.add((order["customer_id"], order["status"]))

print(len(pairs))

A list is not a good key. Python gives an error:

try:
    pairs.add([1, "paid"])
except TypeError as error:
    print(error)

A tuple with a list inside cannot be hashed either. A key must be immutable all the way down. A dict cannot be a key.

To remove duplicate orders, build a tuple from the fields that make the key, as in the first example.

You can skip this section. It is not needed for the next chapter.

A lambda is a small function written on one line. The code `lambda: status` is a function that returns `status`. A closure is a function that uses a name from outside itself.

Here, three lambdas each use the name `status`:

statuses = ["paid", "refunded", "pending"]

checks = [lambda: status for status in statuses]
print([check() for check in checks])

Python looks up that name when you call the function, not when you write it. To keep each value, use a default argument. Python sets a default once, when it makes the function:

checks = []
for status in statuses:
    def check(status=status):
        return status
    checks.append(check)

print([check() for check in checks])

The lambda form of the same fix is shorter:

checks = [lambda status=status: status for status in statuses]
print([check() for check in checks])

Most of these exercises use ideas from this chapter. Before you write code, ask: does anything here share an object?

Try the first three now. The fourth is optional, and you can try it later.