Scaling Python & Interview Capstone

This chapter works one Python interview problem from the first question to a bigger file. You use what you learned before: dicts, generators, Counter, sorting, bad rows, type hints and tests.

The problem: you get a CSV file of orders. Return the top N customers by paid revenue.

Paid revenue is the sum of amount over the orders with status paid. The file is the shop again:

# orders.csv has these six rows below a header line.
# Every value in a CSV file is text.

You work in six steps:

1. Ask questions and say the problem back.
2. Write examples and edge cases, with the answer by hand.
3. Write a simple first version.
4. Test it on the shop rows.
5. Say the cost in plain words.
6. Make it work for a big file.

The last sections give you a script to say aloud and a plan for a 45-minute round.

Do not start with code. First say the problem back in your own words. For example: "You want the top N customers, by the sum of their paid amounts."

Then ask short questions. Each answer removes a guess. These are the questions, with the answers this chapter assumes:

Work the shop rows by hand before you write code. The result is your first test. This table decides, for each order, if it counts:

The totals are: customer 4 has 80.00, customer 1 has 65.00 (40.00 plus 25.00) and customer 3 has 15.50. Customer 2 has no paid order. So the top 3 are customers 4, 1 and 3.

Now list the edge cases. An edge case is an input at the border of what you expected:

• An empty amount. Skip it on a pending order. Reject it on a paid order.
• A refunded order. It has an amount (60.00), but it does not count.
• An unknown customer, such as customer 9. We count by customer_id, so we do not need the customers table.
• A customer with no orders, such as Omar (customer 5). He does not appear.
• An empty file. The answer is an empty list.

Write the simplest code that is correct. The idea has three parts: add up the paid amounts for each customer, sort the totals, and take the first N.

from collections import Counter

def top_customers(rows: list[dict[str, str]], n: int) -> list[tuple[str, float]]:
    totals: Counter[str] = Counter()
    for row in rows:
        if row["status"] == "paid":
            totals[row["customer_id"]] += float(row["amount"])
    ranked = sorted(totals.items(), key=lambda item: (-item[1], int(item[0])))
    return ranked[:n]

Read it in four steps:

1. Counter gives 0 to a customer it has not seen, so += works with no check.
2. The status test skips refunded and pending orders.
3. The key (-item[1], int(item[0])) sorts by total, biggest first. Equal totals sort by customer_id as a number.
4. [:n] keeps the first N items.

Run the function on the six shop rows. The text below is the CSV file. csv.DictReader turns each line into a dict:

import csv, io

SHOP = """order_id,customer_id,status,amount,order_date
101,1,paid,40.00,2026-01-03
102,1,paid,25.00,2026-01-10
103,2,refunded,60.00,2026-01-11
104,3,paid,15.50,2026-01-15
105,3,pending,,2026-01-20
106,4,paid,80.00,2026-01-21
"""
rows = list(csv.DictReader(io.StringIO(SHOP)))

print(top_customers(rows, 3))
print(top_customers(rows, 2))
print(top_customers([], 3))
print(top_customers(rows, 10))

Order 105 did not crash the code. Its status is pending, so the loop skips it before float("") runs.

The output shows 80.0, not 80.00, because a float drops the last zero. A later section fixes that.

In a real project you keep these checks as tests. A test is a function that fails if the answer is wrong:

def test_top_three():
    assert top_customers(rows, 3) == [("4", 80.0), ("1", 65.0), ("3", 15.5)]

def test_empty_file():
    assert top_customers([], 3) == []

Interviewers ask: how fast is it, and how much memory does it use? Let R be the number of rows and C the number of customers.

Big-O is a short way to write this. Time is O(R + C log C).

The totals need O(C) memory, but the list of rows still needs O(R). Step 6c removes that list.

log C grows very slowly: for a million customers it is only about 20.

Say it in words too: "Time grows with the rows. Memory grows with the customers."

One small improvement: if C is huge and N is small, heapq.nlargest(n, ...) finds the top N without sorting every total. Mention it. Do not code it first.

Its key must still put the smaller id first on ties.

A float stores many decimal numbers a little wrong. Decimal stores them exactly.

You met Decimal in the chapter Types, Time, Nulls & Numeric Correctness. Compare the two:

from decimal import Decimal

print(0.1 + 0.2)
print(Decimal("0.10") + Decimal("0.20"))

Change two things in the first version: float becomes Decimal, and the return type changes. The rest stays the same. Run it again on the same rows:

from decimal import Decimal

def top_customers(rows: list[dict[str, str]], n: int) -> list[tuple[str, Decimal]]:
    totals: Counter[str] = Counter()
    for row in rows:
        if row["status"] == "paid":
            totals[row["customer_id"]] += Decimal(row["amount"])
    ranked = sorted(totals.items(), key=lambda item: (-item[1], int(item[0])))
    return ranked[:n]

print(top_customers(rows, 3))

A real file has bad rows. With the Decimal version, one paid order with an empty amount stops the run: Decimal("") raises InvalidOperation. You need a check, a count and a limit.

The check raises ValueError for one bad row, as in the chapter Validation, Contracts & Error Handling. The loop counts the bad row and goes on. At the end, the limit decides if the whole run is bad:

from collections import Counter
from collections.abc import Iterable
from decimal import Decimal, InvalidOperation

def check(row: dict[str, str]) -> tuple[str, Decimal]:
    if not row["customer_id"]:
        raise ValueError("customer_id is missing")
    try:
        amount = Decimal(row["amount"])
    except InvalidOperation:
        raise ValueError("amount is not a number")
    if not amount.is_finite() or amount < 0:
        raise ValueError("amount must be 0 or more")
    return row["customer_id"], amount

def top_customers(
    rows: Iterable[tuple[int, dict[str, str]]], n: int, max_reject: float = 0.20
) -> tuple[list[tuple[str, Decimal]], dict[str, object]]:
    totals: Counter[str] = Counter()
    reasons: Counter[str] = Counter()
    first_bad: list[tuple[int, str]] = []
    seen = 0
    for line, row in rows:
        seen += 1
        if row["status"].strip().lower() != "paid":
            continue
        try:
            customer_id, amount = check(row)
        except ValueError as error:
            reasons[str(error)] += 1
            if len(first_bad) < 3:
                first_bad.append((line, str(error)))
            continue
        totals[customer_id] += amount
    rejected = sum(reasons.values())
    if seen and rejected / seen > max_reject:
        raise ValueError(f"too many bad rows: {rejected} of {seen}")
    ranked = sorted(totals.items(), key=lambda item: (-item[1], int(item[0])))
    report = {"seen": seen, "rejected": rejected, "first_bad": first_bad}
    return ranked[:n], report

The status test now ignores spaces and capital letters, so " Paid" also counts.

Now add four rows to the shop file. Rows 107 and 108 are the first run. Rows 109 and 110 make the second run worse:

def numbered(text: str):
    return enumerate(csv.DictReader(io.StringIO(text)), start=2)

extra = "107,9,paid,12.00,2026-01-22\n108,,paid,9.00,2026-01-23\n"
top, report = top_customers(numbered(SHOP + extra), 3)
print(top)
print(report)

worse = extra + "109,2,paid,twelve,2026-01-24\n110,5,paid,,2026-01-25\n"
try:
    top_customers(numbered(SHOP + worse), 3)
except ValueError as error:
    print(error)

So far the rows sat in a list. A big file does not fit in memory. A generator reads one row at a time and forgets it.

The function above already accepts a stream, because its loop reads one pair at a time.

This code writes a test file of 100,000 orders. Then it streams the file through top_customers:

def read_orders(path: str):
    with open(path, newline="", encoding="utf-8") as file:
        yield from enumerate(csv.DictReader(file), start=2)

with open("big_orders.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["order_id", "customer_id", "status", "amount", "order_date"])
    for i in range(1, 100_001):
        status = "paid" if i % 4 else "refunded"
        writer.writerow([i, (i % 1000) + 1, status, f"{(i % 50) + 1}.00", "2026-01-01"])

top, report = top_customers(read_orders("big_orders.csv"), 3)
print(top)
print(report)

Now see the memory. tracemalloc measures the most memory Python used. Compare the stream with a list of all the rows. Your numbers will be close, not the same:

import tracemalloc

def peak_mb(make) -> float:
    tracemalloc.start()
    make()
    peak = tracemalloc.get_traced_memory()[1]
    tracemalloc.stop()
    return round(peak / 1_000_000, 1)

def stream():
    for line, row in read_orders("big_orders.csv"):
        pass

print("stream", peak_mb(stream), "MB")
print("list  ", peak_mb(lambda: list(read_orders("big_orders.csv"))), "MB")

Sometimes you must write results too. Write them in batches of a fixed size, not in one big list. Python 3.12 has itertools.batched for this:

from itertools import batched

print(list(batched(range(1, 8), 3)))

To write in batches, loop over them. Here rows is any stream of output rows and writer is a csv writer:

for batch in batched(rows, 1000):
    writer.writerows(batch)

Streaming is not enough. Ask of every variable: what makes it grow? This table does it for our code:

Look at first_bad. A list of every bad row would grow with the bad rows.

A file with millions of bad rows would fill memory. So the code keeps three examples and a count.

Do not guess where the time goes. Time each part, then change only the slowest part.

This code times reading alone, and then reading plus adding up:

from time import perf_counter

start = perf_counter()
for line, row in read_orders("big_orders.csv"):
    pass
read_seconds = perf_counter() - start

start = perf_counter()
top_customers(read_orders("big_orders.csv"), 3)
total_seconds = perf_counter() - start

print(f"read only: {read_seconds:.2f} s")
print(f"read and add up: {total_seconds:.2f} s")

If reading alone is most of the time, faster Python code will not help much. The file is the slow part. If the gap is large, look at check and the sums.

Two standard tools help more. timeit runs a small piece of code many times to compare versions.

cProfile shows which function takes the most time in a full run.

A slow program is slow in one place: the processor, the memory, the disk or the network. Each one has a different first move:

Python has a lock called the GIL (global interpreter lock). It lets only one thread run Python code at a time.

So threads do not speed up a CPU-heavy loop. The chapter Concurrency, Parallelism & Performance explains more.

A process is a separate running copy of Python. I/O means reading and writing (input and output).

Pure Python is not always the answer. For our problem, the same result in SQL is short. This query runs on the shop orders:

SELECT customer_id, SUM(amount) AS revenue
FROM orders
WHERE status = 'paid'
GROUP BY customer_id
ORDER BY revenue DESC, customer_id
LIMIT 3;

Here is when each tool fits:

A quick reminder from the Spark chapters: Spark splits a big table into partitions and works on them on many machines. A shuffle moves rows between machines, and it is the costly step.

Speak while you work. This script is a model. Use your own words, and keep each sentence short:

1. "Let me say the problem back: the top N customers by the sum of paid amounts."
2. "I have questions: how big is the file, what counts as paid, and what is a bad row?"
3. "By hand, the answer for the shop rows is customer 4, then 1, then 3."
4. "First I write a simple version with Counter and sorted. Then I test it."
5. "Next I handle bad rows: I skip them, count them, and stop above 20%."
6. "The time grows with the rows. The memory grows with the customers."
7. "If the file grows, I stream it. If it is in a database, I use SQL. If it no longer fits one machine, I would think about Spark."

This is one plan, not a rule. Rounds differ, so ask the interviewer how they use the time. Practise it with a timer:

The total is 45 minutes. If time runs short, stop adding and finish a correct simple version. A working simple answer is a better place to stop than a half-finished fast one.

Many data engineering interview loops have a Python round. Prep guides and candidate reports describe it as live coding or a take-home task.

A live round is often 45 to 60 minutes. Companies differ, so ask the recruiter what your round is.

These are the kinds of problems that people report. The first column is the kind, and the second column gives examples:

Guides often describe a short warm-up first and one bigger problem second. Do not count on it. The same guides say that the bigger problem looks like real pipeline work more often than like a puzzle contest.

Interviewers are reported to watch for these habits. They come from prep guides, not from official company rules, so use them as a good default:

• Say the edge cases before you code: empty input, one row, a missing key, a bad row.
• Use plain Python first. Ask before you use pandas.
• Write type hints on the function signature.
• Use a generator or a single pass when the input is large. Do not call list() on a big input.
• Get a correct simple answer first. Make it faster after that.

The same guides say that some topics are asked less often for data engineers. These are trees, linked lists, dynamic programming and backtracking.

Hash maps, strings, arrays, sliding windows and heaps are asked more often. Some companies still ask the rare topics, so check reports for your company before you skip one.

On this site, the Algorithm patterns path covers the light algorithm problems with data engineering stories. The Python interview questions cover the warm-up basics and the project questions.

Each exercise opens in the Python workspace. They are harder than the chapter problem.

In loop week, pick one exercise after you can work the chapter problem in 30 minutes. Set a timer for 20 minutes. Talk aloud as you work.

You worked one problem the way an interview round works. These are the habits to keep:

• Ask first, then write. Work the answer by hand.
• Start simple: a Counter, a sorted call with a key, and type hints.
• Keep tests that match your hand answer.
• Use Decimal for money. Count bad rows and set a limit.
• Stream with a generator, and know what grows with the input.
• Measure first. Move to SQL or Spark only when the data needs it.