Python For Data Engineering Mindset

In a data engineering Python interview, you usually get a list of records. You group them, count them, remove duplicates or join them.

It is not a puzzle. It is the daily work of the job.

This chapter teaches the way of thinking behind that work. Every job has the same shape: records go in, a function changes them, and records come out.

The examples use the same shop as the SQL chapters: customers, orders and order_items.

In SQL, one line of a table is a row. In Python, we call it a record. SQL NULL is Python None, and both mean "no value".

You do not need to know Python yet. Each new word is explained the first time you see it. In this chapter, you learn to:

1. Hold the shop as a list of records, and read one value from a record.
2. Loop over the records, and keep only the ones you want.
3. Write a function that turns a record into a new record.
4. Write a contract: a check on every record that enters your code.
5. Choose what to do with a bad record. A bad record is one with a missing field or an unusable value.
6. Know when to use SQL, plain Python or pandas. pandas is a Python library, which means a set of ready-made tools, for tables.

A variable is a name for a value. The sign = stores a value under a name. So order = ... means "call this value order".

A dict (short for dictionary) is a group of labelled values. Each label is a key, and each key has one value. In a dict, a colon links a key to its value.

The function print(x) shows the value of x. The "Printed" table under each example is the text Python shows. Here is order 101 as a dict:

order = {
    "order_id": 101,
    "customer_id": 1,
    "status": "paid",
    "amount": 40.00,
    "order_date": "2026-01-03",
}

print(order["status"])
print(order["amount"])

Square brackets read one value from a dict by its key. Python prints 40.00 as 40.0, because it is a number, not text.

A list holds many values in order. A list of dicts is like a table: each dict is one record.

Square brackets also read one value from a list: orders[0] is the first record. This is the shop's orders:

orders = [
    {"order_id": 101, "customer_id": 1, "status": "paid",
     "amount": 40.00, "order_date": "2026-01-03"},
    {"order_id": 102, "customer_id": 1, "status": "paid",
     "amount": 25.00, "order_date": "2026-01-10"},
    {"order_id": 103, "customer_id": 2, "status": "refunded",
     "amount": 60.00, "order_date": "2026-01-11"},
    {"order_id": 104, "customer_id": 3, "status": "paid",
     "amount": 15.50, "order_date": "2026-01-15"},
    {"order_id": 105, "customer_id": 3, "status": "pending",
     "amount": None, "order_date": "2026-01-20"},
    {"order_id": 106, "customer_id": 4, "status": "paid",
     "amount": 80.00, "order_date": "2026-01-21"},
]

print(len(orders))
print(orders[0]["order_id"])
print(orders[4])

Python prints a dict with single quotes. Order 105 has no amount, so its amount is None.

Now the other two tables. Each has its own grain, which is the thing one record stands for.

A customer record stands for one customer. An order item record stands for one product on one order.

customers = [
    {"customer_id": 1, "name": "Asha", "country": "IN"},
    {"customer_id": 2, "name": "Ben", "country": "GB"},
    {"customer_id": 3, "name": "Carla", "country": "US"},
    {"customer_id": 4, "name": "Dev", "country": None},
    {"customer_id": 5, "name": "Omar", "country": "IN"},
]

order_items = [
    {"order_id": 101, "product": "Pen", "quantity": 2},
    {"order_id": 101, "product": "Notebook", "quantity": 1},
    {"order_id": 102, "product": "Bag", "quantity": 1},
    {"order_id": 103, "product": "Lamp", "quantity": 1},
    {"order_id": 104, "product": "Pen", "quantity": 3},
    {"order_id": 106, "product": "Chair", "quantity": 1},
    {"order_id": 106, "product": "Lamp", "quantity": 1},
]

print(len(customers))
print(len(order_items))

A loop repeats steps for every item in a list. An if line runs its steps only when its test is true. The steps that belong to a loop or an if are indented, which means pushed to the right.

This loop counts the paid orders and adds up their amounts. It uses the orders list from the last section:

paid_count = 0
paid_total = 0

for order in orders:
    if order["status"] == "paid":
        paid_count = paid_count + 1
        paid_total = paid_total + order["amount"]

print(paid_count, paid_total)

The call print with commas puts a space between the values.

Two equals signs, ==, ask "are these the same?". One equals sign stores a value. So paid_count = paid_count + 1 means: take the old count, add 1, and store it back.

This table shows the loop, one record at a time:

The if skips orders 103 and 105. So the None amount of order 105 is never added, and the loop does not break.

In SQL, this was WHERE status = 'paid' with COUNT and SUM. The idea is the same: keep some records, then count and add.

A function is a named set of steps. It takes inputs and gives back a result with return. A transform is a function that takes a record and returns a new record.

This transform keeps order_id and adds a new field, is_paid. It is True when the status is "paid". The loop then runs it on every order:

def to_summary(order):
    return {
        "order_id": order["order_id"],
        "is_paid": order["status"] == "paid",
    }

summaries = []
for order in orders:
    summaries.append(to_summary(order))

print(summaries[0])
print(summaries[4])
print(len(summaries))

Six order records went in. Six new summary records came out. The orders list did not change.

That last point matters. A function gets the same dict, not a copy. If the function changes it, the caller's dict changes too.

Jobs run again after a failure, so the same order can go through a function twice. In this function, order["amount"] = ... stores a new value under the key "amount". Call it twice on one order:

order_101 = {"order_id": 101, "amount": 40.0}

def add_fee_bad(order):
    order["amount"] = order["amount"] + 1.0
    return order

add_fee_bad(order_101)
add_fee_bad(order_101)
print(order_101)

The fix is to build a new dict and return it. A function that only uses its inputs, and changes nothing else, is called pure. Calling it twice gives the same answer:

order_102 = {"order_id": 102, "amount": 25.0}

def add_fee(order):
    return {"order_id": order["order_id"], "amount": order["amount"] + 1.0}

first = add_fee(order_102)
second = add_fee(order_102)
print(first)
print(second)
print(order_102)

A boundary is the place where data enters your code, such as a file or a web service. A contract is a promise about the records at that place. It says which fields a record needs, and which values are allowed.

Without a contract, the error message does not say what is wrong with the record. In a real job, it can also appear many steps later. Try to add every amount in the shop.

The try line runs its steps. If an error of the named kind (TypeError) happens, the except steps run instead. The words as error give the error the name error:

total = 0
try:
    for order in orders:
        total = total + order["amount"]
except TypeError as error:
    print("stopped at order", order["order_id"])
    print(error)

Order 105 is pending, so a missing amount is normal. The loop did not know that. A contract says it in one place.

This contract has two rules:

• Every order has the five fields.
• A paid order has an amount.

The function check_order raises an error when a rule breaks. Raise means: stop with an error and a message. Here is the function:

REQUIRED = ["order_id", "customer_id", "status", "amount", "order_date"]

def check_order(order):
    missing = []
    for field in REQUIRED:
        if field not in order:
            missing.append(field)
    if missing:
        raise ValueError(f"order {order.get('order_id')}: missing {missing}")
    if order["status"] == "paid" and order["amount"] is None:
        raise ValueError(f"order {order['order_id']}: paid order has no amount")

Read the function with these notes:

• field not in order is true when the key is missing.
• if missing: is true when the list has items.
• is None tests for None.
• ValueError is the kind of error for a wrong value.
• f"..." puts the value of each {name} into the text.
• order.get("order_id") gives None if the key is missing, instead of an error.

Now test it on three records. Order 105 is from the shop. Orders 107 and 108 are two new bad records:

bad_paid = {
    "order_id": 107,
    "customer_id": 2,
    "status": "paid",
    "amount": None,
    "order_date": "2026-01-22",
}
incomplete = {"order_id": 108, "status": "paid"}

for record in [orders[4], bad_paid, incomplete]:
    try:
        check_order(record)
        print(record["order_id"], "ok")
    except ValueError as error:
        print(error)

The output contract is the same promise for what you return. Every record out has the same fields, like to_summary above.

Some records will always be bad. You must choose what happens to them. There are three choices:

Quarantine works well when a few records are bad. This loop uses check_order, bad_paid and incomplete from the last section.

It puts each record in a good list or a rejected list. The call str(error) turns the error into its message text:

raw = orders + [bad_paid, incomplete]

good = []
rejected = []
for record in raw:
    try:
        check_order(record)
        good.append(record)
    except ValueError as error:
        rejected.append({"order_id": record.get("order_id"), "reason": str(error)})

print("kept", len(good), "rejected", len(rejected))
for item in rejected:
    print(item)

No record disappeared. You can count the rejected ones and show them to the team that owns the data. You can also get a message if the number grows fast.

Fail loud when the whole input looks wrong. An empty list from a feed that normally has orders is one case.

Writing an empty result over yesterday's good data is worse than stopping. This function stops instead:

def load_orders(records):
    if len(records) == 0:
        raise RuntimeError("no orders received")
    return records

try:
    load_orders([])
except RuntimeError as error:
    print("stopped:", error)

The same question can be answered in SQL, in plain Python, or with a library such as pandas. Here is the question: what is the paid revenue for each customer?

In plain Python, a dict holds one total per customer. It starts empty. For each paid order, the loop reads the old total, adds the amount, and stores the new total:

totals = {}
for order in orders:
    if order["status"] == "paid":
        customer = order["customer_id"]
        totals[customer] = totals.get(customer, 0) + order["amount"]

print(totals)

Customer 1 has two paid orders: 40 + 25 = 65. Customers 3 and 4 have one each. Customer 2 only has a refund, and customer 5 has no orders, so neither appears.

In SQL, this is SUM(amount) with WHERE status = 'paid' and GROUP BY customer_id. The logic is the same, and only the tool is different.

pandas is a Python library, a set of ready-made tools, for tables. Spark splits big data over many computers. This table shows when to use each tool:

Spark has its own module later in this course. pandas has no chapter here, but the Python interview questions cover it. Both change the tool, not the thinking: records in, a transform, records out.

Interviewers often give a list of dicts and ask for a group, a count or a join.

Start with plain Python. You show each step, and you need no library. Then say how pandas would do it.

These are the words of this chapter:

Each exercise opens in the Python workspace. You write a function, run it on the sample records, and check the result.

The third fits this chapter best. The first needs string methods that a later chapter teaches. The second and fourth also need later chapters, so try them after those.