Testing, Debugging, Logging & Operability

Openingloading the content…

Loadingone moment…

01 · Orientation

What You'll Master Here

Testing proves expected behavior before the run; observability proves what happened during the run.

⏱ 4 min · Topic 1 of 16

Advanced Python data engineering is not just writing transforms. It is proving they work, explaining failures, and giving operators enough evidence to trust a run.

This chapter connects pytest-style tests, fixtures, golden outputs, failure-path assertions, dependency mocking, row-count debugging, structured logging, metrics, and run manifests.

The mindset is simple: every important promise in the pipeline should have a small test, a clear log, a health signal, or a receipt.

Core mental model

Testing proves expected behavior before the run; observability proves what happened during the run.

Why data engineers care

The difference between a script and a production data job is whether someone can debug and rerun it at 2 AM without guessing.

fixture: A reusable test input or setup object that makes tests small and readable.
golden output: A known expected output used to prove a transform contract has not drifted.
structured log: A log event with named fields such as run_id, partition, rows_seen, and status.
operability: The practical ability to run, monitor, debug, and rerun a job safely.

Data job test pyramid

layer	example	confidence
unit	normalize_order fixture rows	fast rule proof
contract	golden accepted/rejected outputs	schema and evidence proof
integration	temp files and mocked API/db boundary	boundary proof

Common mistake

Testing only the happy path. Rejected rows, warnings, and broken dependencies fail first in production.

Better habit

Test pure transforms with tiny input/output examples.

Assert failure evidence, not just that an exception happened.

Log run context and emit manifests that reconcile counts.

Senior signal

Say: I would test accepted and rejected outputs, mock unstable boundaries, log structured run context, and emit a manifest with counts that reconcile.

Remember this

Reliability is a product feature of the pipeline.

02 · Testing

Testing Is Pipeline Evidence, Not Just Code Coverage

input, rule, output, evidence.

⏱ 5 min · Topic 2 of 16

A data pipeline test should prove a contract: given these source rows, these accepted rows, rejected rows, warnings, and manifest counts are produced.

Coverage numbers do not tell you whether the business key, row grain, or failure path is right. Expected output examples do.

Use tests as executable documentation. A teammate should understand the rule by reading the fixture and expected output.

Core mental model

Every test should answer: input, rule, output, evidence.

Why data engineers care

A tested transform is easier to change because the expected data movement is locked down.

A pytest-style contract testworked example

Python

def test_normalize_order_accepts_valid_paid_order():
    raw = {"order_id": "1001", "amount": "12.30", "status": "paid"}

    actual = normalize_order(raw)

    assert actual == {
        "order_id": "1001",
        "amount": Decimal("12.30"),
        "status": "paid",
    }

Result · 1 row

input	expected output	contract proven
valid paid order	NormalizedOrder	amount parsed and status preserved

Common mistake

Testing implementation details instead of output contracts. Refactors break tests even when behavior is still correct.

Better habit

Name tests by behavior.

Use tiny rows.

Assert exact output shape.

Test naming

A good test name reads like a rule: test_rejects_missing_amount_with_source_context.

Remember this

Data tests should prove visible data behavior.

03 · Unit tests

Unit Tests For Pure Transforms

Pure input plus config should produce deterministic output or deterministic rejection.

⏱ 4 min · Topic 3 of 16

Pure transforms are the easiest code to test because they do not read files, call APIs, or depend on time.

Give the function one raw record and assert the exact normalized record or rejected evidence.

If a transform is hard to unit test, it probably owns too many responsibilities.

Core mental model

Pure input plus config should produce deterministic output or deterministic rejection.

Why data engineers care

Fast unit tests give confidence before a job touches external systems.

Test accepted and rejected transform pathsworked example

Python

def test_normalize_order_rejects_missing_amount():
    raw = {"line_number": 3, "order_id": "1002", "amount": "", "status": "paid"}

    result = normalize_order_or_reject(raw)

    assert result == {
        "kind": "rejected",
        "line_number": 3,
        "field": "amount",
        "reason": "amount is required",
    }

Result · 1 row

kind	line_number	field	reason
rejected	3	amount	amount is required

Common mistake

Testing through the file reader for every row rule. Tests become slow and hide the rule under I/O setup.

Better habit

Keep pure row rules in pure functions.

Assert exact rejected rows.

Avoid real clocks/files/network in unit tests.

Design feedback

If a unit test needs a real file just to test amount parsing, move parsing into a smaller pure helper.

Remember this

Pure transforms should be boring to test.

04 · Practice

Writing A Test For A Transform, Start To Finish

the ordinary one, the empty one, the duplicate one, and the one that should be rejected.

⏱ 6 min · Topic 4 of 16

The chapter has argued that tests are evidence. This section writes one, so the argument has a shape you can copy.

The transform is ordinary: group order rows by buyer and total the amounts. The test is four cases, and choosing those four is the actual skill — the happy path is the least interesting of them.

Notice what the test does not do. It does not read a file, it does not call a database, and it does not assert on a log line. It is a pure function tested with literal inputs, which is why it runs in milliseconds and never flakes.

Core mental model

Four cases: the ordinary one, the empty one, the duplicate one, and the one that should be rejected.

Why data engineers care

A test suite of happy paths tells you the code runs. A suite with the empty case, the duplicate case and the bad-value case tells you what the code promises when the data misbehaves — which is the only time anybody reads it.

pure transform: A function whose output depends only on its input. The cheapest thing in the world to test.
rejected rows: Bad input returned as data rather than raised, so a test can assert on it and a run can report it.

The transform, kept pureworked example

Python

from collections import defaultdict
from decimal import Decimal

# Sum amounts per buyer. Rows with an unparseable amount are rejected, not raised.
def total_by_buyer(rows):
    totals, rejected = defaultdict(Decimal), []
    for row in rows:
        try:
            amount = Decimal(str(row["amount"]))
        except Exception:
            rejected.append(row)
            continue
        totals[row["buyer"]] += amount
    return dict(totals), rejected

It takes rows and returns rows. No file, no clock, no client — so the test needs no fixtures beyond literals, and the rejected list makes the failure path assertable.

The four casesworked example

Python

def test_totals_by_buyer():
    rows = [
        {"buyer": "amir", "amount": "10.00"},
        {"buyer": "eve", "amount": "5.50"},
        {"buyer": "amir", "amount": "2.50"},     # duplicate buyer: must sum
    ]
    totals, rejected = total_by_buyer(rows)
    assert totals == {"amir": Decimal("12.50"), "eve": Decimal("5.50")}
    assert rejected == []

def test_empty_input_is_not_an_error():
    assert total_by_buyer([]) == ({}, [])

def test_bad_amount_is_rejected_not_raised():
    bad = {"buyer": "cara", "amount": "12,5"}
    totals, rejected = total_by_buyer([bad])
    assert totals == {}
    assert rejected == [bad]                     # reported, not swallowed

def test_amounts_are_exact():
    rows = [{"buyer": "a", "amount": "0.1"}] * 3
    totals, _ = total_by_buyer(rows)
    assert totals["a"] == Decimal("0.3")         # would fail with floats

Each test names one promise. The last one is the reason `Decimal` is in the transform at all, and it would fail the moment somebody "simplified" it to `float`.

Which four cases

Case	What it pins	What breaks without it
Ordinary input, with a repeat	The aggregation itself	Grouping regressions
Empty input	Empty is a valid answer, not an error	A crash on the first quiet day
Unparseable value	Rejection is reported, not raised	One bad row kills the batch
Exactness / boundary	The reason for the design choice	A "simplification" that quietly loses pennies

Common mistake

Testing only the happy path. The suite proves the code runs and says nothing about what it promises when the data is wrong — which is the case you wrote the rejection logic for.

Asserting on the whole output dict when only one field matters. The test fails whenever an unrelated field is added, so people stop reading its failures and start updating it.

Testing through a file or a database because "that is realistic". It is slower, it flakes, and it tests the I/O rather than the logic. Keep the transform pure and test the boundary separately.

Better habit

One promise per test, and name the test after the promise.

Write the empty case and the bad-value case before the happy one.

Return rejections as data so the failure path is assertable.

Interview note

"How would you test this?" is answered with the four cases, not with a framework. Naming the empty case and the rejection case unprompted is what separates a considered answer from a generic one.

Watch out

The platform's Python sandbox has no `pytest`, so the exercises test transforms through their graded cases rather than through a test file. The four-case habit is the transferable part.

Remember this

Keep the transform pure, return rejections as data, and write four cases: ordinary, empty, rejected, and the one that justifies your design.

05 · Fixtures

Fixture Design For Data Engineering

Start from a valid record; override the one thing the test cares about.

⏱ 5 min · Topic 5 of 16

Fixtures keep test data readable. A good fixture contains the minimum valid row, then tests override only the field being exercised.

Avoid giant realistic payloads for every test. Use one small canonical row plus targeted edge cases.

For data engineering, fixture families often include valid row, missing required field, malformed value, duplicate key batch, and late event batch.

Core mental model

Start from a valid record; override the one thing the test cares about.

Why data engineers care

Readable fixtures make tests explain the business rule instead of burying it in setup.

Fixture factory with overridesworked example

Python

def raw_order(**overrides):
    row = {
        "file_name": "orders.csv",
        "line_number": 2,
        "order_id": "1001",
        "amount": "12.30",
        "status": "paid",
    }
    row.update(overrides)
    return row

bad_amount = raw_order(line_number=3, amount="bad")

Result · 2 rows

fixture	line_number	amount	purpose
raw_order()	2	12.30	valid default
raw_order(amount="bad")	3	bad	invalid amount case

Common mistake

Copying a 40-field payload into every test. The important field disappears in noise and changes become tedious.

Better habit

Use minimal valid defaults.

Override one or two fields per test.

Keep fixture values realistic enough to reveal type issues.

Fixture discipline

Small does not mean fake-looking. Use realistic ids, amounts, timestamps, and line numbers, but only the fields the rule needs.

Remember this

Fixtures are teaching tools for future maintainers.

06 · Golden tests

Golden Output Tests

Freeze the visible contract, not the internal implementation.

⏱ 5 min · Topic 6 of 16

A golden output test compares actual transform output to a known expected result.

For data jobs, golden outputs are especially useful when a transform returns multiple artifacts: accepted rows, rejected rows, warnings, and manifest counts.

Keep golden examples small enough to inspect. A good golden test is a story, not a dump.

Core mental model

Freeze the visible contract, not the internal implementation.

Why data engineers care

Golden tests catch accidental contract drift in field names, ordering, reasons, and counts.

Golden expected output for a tiny batchworked example

Python

expected = {
    "accepted": [{"order_id": "1001", "amount": Decimal("12.30")}],
    "rejected": [{"line_number": 3, "field": "amount", "reason": "amount is required"}],
    "manifest": {"rows_seen": 2, "accepted": 1, "rejected": 1},
}

assert transform_batch(rows) == expected

Result · 3 rows

artifact	expected
accepted	1 normalized row
rejected	1 rejected row with reason
manifest	rows_seen=2, accepted=1, rejected=1

Common mistake

Letting output row order vary in golden tests. Tests fail randomly or hide nondeterministic output.

Better habit

Sort outputs deterministically.

Keep golden fixtures tiny.

Assert rejected reasons exactly.

What to say

I would use golden tests for tiny representative batches so the accepted output, rejected output, and manifest all stay locked.

Remember this

Golden tests protect the pipeline contract users actually consume.

07 · Failure paths

Testing Rejected Rows And Failure Paths

Every failure path should be as deterministic as an accepted row.

⏱ 4 min · Topic 7 of 16

Rejected rows are not side effects. They are outputs, and they deserve tests.

Test missing fields, malformed values, unknown enum values, duplicate keys, fatal schemas, warning-only cases, and threshold failures.

A good failure-path test asserts source context as well as reason text.

Core mental model

Every failure path should be as deterministic as an accepted row.

Why data engineers care

Most production incidents happen in paths that were not part of the happy-path demo.

Assert rejection evidence, not just failureworked example

Python

def test_rejected_row_has_source_context():
    row = raw_order(file_name="orders.csv", line_number=7, amount="bad")

    rejected = validate_or_reject(row)

    assert rejected["file_name"] == "orders.csv"
    assert rejected["line_number"] == 7
    assert rejected["field"] == "amount"
    assert rejected["reason"] == "amount must be decimal"

Result · 1 row

file_name	line_number	field	reason
orders.csv	7	amount	amount must be decimal

Common mistake

Testing only that ValueError was raised. The job may lose file, line, field, or reason context without test failure.

Better habit

Assert source context.

Assert warning and fatal paths separately.

Test thresholds around boundary values.

Evidence quality

A failed test should tell you exactly which evidence field regressed.

Remember this

Failure-path tests make bad data safe and explainable.

08 · Mocks

Mocking Clocks, Files, APIs, And Databases

Real dependencies at the edge; fake dependencies in tests; pure logic in the middle.

⏱ 5 min · Topic 8 of 16

Mocks are most valuable at unstable boundaries: current time, external APIs, databases, file systems, and random ids.

Do not mock pure logic. Feed pure logic simple data.

When possible, dependency injection is cleaner than patching: pass a clock function, a client, or a writer into the code that needs it.

Core mental model

Real dependencies at the edge; fake dependencies in tests; pure logic in the middle.

Why data engineers care

Stable tests need stable dependencies. Production dependencies are not stable enough for unit tests.

Inject a fixed clockworked example

Python

from datetime import datetime, timezone

def build_manifest(rows_seen, accepted, rejected, now):
    return {
        "created_at": now().isoformat(),
        "rows_seen": rows_seen,
        "accepted": accepted,
        "rejected": rejected,
    }

fixed_now = lambda: datetime(2026, 6, 19, 12, 0, tzinfo=timezone.utc)
manifest = build_manifest(3, 2, 1, now=fixed_now)

Result · 1 row

created_at	rows_seen	accepted	rejected
2026-06-19T12:00:00+00:00	3	2	1

Common mistake

Mocking the function under test instead of its dependency. The test proves the mock, not the behavior.

Better habit

Inject clocks and clients where practical.

Patch only boundary calls.

Assert calls and outputs when mocking writers.

unittest.mock

Python’s unittest.mock is useful for patching boundary dependencies, but pure functions usually need no mocks at all.

Remember this

Mock boundaries, not business rules.

09 · Debugging

Debugging With Row Counts And Small Reproductions

Find the first stage where the count changes unexpectedly.

⏱ 5 min · Topic 9 of 16

The fastest data debugging path is usually row counts by stage: source, parsed, accepted, rejected, written.

When counts diverge, reproduce the issue with the smallest row set that still fails.

This habit prevents random print debugging across a giant file.

Core mental model

Find the first stage where the count changes unexpectedly.

Why data engineers care

Most data bugs are population bugs before they are code bugs.

Row-count checkpoint tableworked example

Python

debug_counts = {
    "source_rows": 1000,
    "parsed_rows": 997,
    "accepted_rows": 960,
    "rejected_rows": 37,
    "written_rows": 960,
}

Result · 4 rows

stage	count	signal
source_rows	1000	source delivered
parsed_rows	997	3 parse failures
accepted_rows	960	37 validation rejects
written_rows	960	write reconciles

Common mistake

Inspecting final bad output before checking stage counts. You may debug the wrong stage for hours.

Better habit

Add stage counts early.

Reproduce with three to five rows.

Compare accepted count to written count.

Debug ladder

Count first, inspect second. Counts tell you which layer deserves inspection.

Remember this

Row counts turn debugging from a search into a narrowing process.

10 · Logging

Structured Logging With `logging.getLogger(name)`

A log event should say what happened, where it happened, and which run it belongs to.

⏱ 5 min · Topic 10 of 16

Logging is for human and machine reading during a run. Use named loggers so logs identify the module that produced them.

Structured context makes logs searchable: run_id, batch_id, partition, file name, row counts, and status.

Do not log sensitive payloads. Log identifiers, counts, and reasons.

Core mental model

A log event should say what happened, where it happened, and which run it belongs to.

Why data engineers care

When a scheduled job fails, logs are often the first artifact an operator reads.

Named logger with run contextworked example

Python

import logging

logger = logging.getLogger(__name__)

logger.info(
    "orders_batch_validated",
    extra={
        "run_id": "orders-2026-06-19T12",
        "partition": "event_day=2026-06-19",
        "rows_seen": 1000,
        "accepted": 960,
        "rejected": 40,
    },
)

Result · 4 rows

key	value
run_id	orders-2026-06-19T12
partition	event_day=2026-06-19
accepted	960
rejected	40

Common mistake

Logging only "job failed". Operators still need to hunt for batch, partition, and stage context.

Better habit

Use module loggers.

Attach stable run context.

Avoid logging raw sensitive values.

Python logging

logging.getLogger(__name__) creates a module-scoped logger that works with the application’s logging configuration.

Remember this

Logs should be searchable run evidence, not loose narration.

11 · Context

Log Context: Batch ID, File, Partition, Row Counts

Run context is the join key across operational evidence.

⏱ 4 min · Topic 11 of 16

A log without context is a sentence floating in space. Context connects it to a run, file, partition, and count trail.

The same context should appear in logs, manifests, and metrics so debugging can move across tools.

Use consistent field names. If the manifest says run_id, the logs should not say job_run_identifier.

Core mental model

Run context is the join key across operational evidence.

Why data engineers care

Consistent context makes incidents traceable across logs, files, and dashboards.

Reusable context dictionaryworked example

Python

context = {
    "run_id": "orders-2026-06-19T12",
    "batch_id": "orders-2026-06-19",
    "source_file": "orders.csv",
    "partition": "event_day=2026-06-19",
}

logger.info("write_completed", extra={**context, "written_rows": 960})

Result · 4 rows

field	why it exists
run_id	correlates one execution
source_file	locates input
partition	locates output
written_rows	reconciles output

Common mistake

Changing log key names across modules. Searches and dashboards miss part of the run.

Better habit

Define shared context keys.

Pass context through run layers.

Reuse context in manifests.

Traceability

A run_id should let you find logs, output files, rejected rows, metrics, and the manifest.

Remember this

Operational evidence needs common keys.

12 · Metrics

Metrics And Health Signals

Metrics are health checks over time.

⏱ 5 min · Topic 12 of 16

Metrics are numeric signals about pipeline health: freshness, duration, rows seen, rejection rate, retry count, and output row counts.

Logs explain events; metrics show trends. A rejection spike is easier to spot in a metric than in thousands of log lines.

Good metrics are tied to thresholds and action. A number nobody watches is decoration.

Core mental model

Metrics are health checks over time.

Why data engineers care

Metrics reveal drift and degradation before users report wrong data.

Compute simple health metricsworked example

Python

rows_seen = 1000
rejected = 40
duration_ms = 842

health = {
    "rejection_rate": rejected / rows_seen,
    "duration_ms": duration_ms,
    "write_count_match": True,
}

Result · 3 rows

metric	value	interpretation
rejection_rate	0.04	watch for spikes
duration_ms	842	runtime trend
write_count_match	true	accepted rows were written

Common mistake

Alerting on every warning. Alert fatigue makes real incidents easier to miss.

Better habit

Emit counts and rates.

Trend metrics over time.

Alert on actionable thresholds.

Health signal

Freshness and rejection rate are often better early-warning signals than job success/failure alone.

Remember this

Metrics make pipeline health visible across runs.

13 · Receipts

Run Manifests, Audit Receipts, And Traceability

A run manifest is the audit join between source, transform, output, logs, and metrics.

⏱ 5 min · Topic 13 of 16

A manifest is the structured receipt for a run. It should reconcile counts and point to output artifacts.

For production jobs, the manifest often becomes the artifact people trust more than a log line because it is structured, stored, and queryable.

Treat the manifest as the final output of operability.

Core mental model

A run manifest is the audit join between source, transform, output, logs, and metrics.

Why data engineers care

Traceability means you can answer what ran, when, with what input, and what it produced.

Manifest evidence with durationworked example

Python

manifest = {
    "run_id": "orders-2026-06-19T12",
    "status": "success_with_warnings",
    "rows_seen": 1000,
    "accepted": 960,
    "rejected": 40,
    "duration_ms": 842,
}

Result · 1 row

run_id	status	rows_seen	accepted	rejected	duration_ms
orders-2026-06-19T12	success_with_warnings	1000	960	40	842

Common mistake

Keeping manifests only in memory. The receipt disappears after the job ends.

Better habit

Store manifests durably.

Include input and output locations.

Make counts reconcile.

What to say

I would treat the manifest as an audit receipt: it records run id, input, output, counts, status, and duration.

Remember this

Manifests turn a run into a traceable event.

14 · Checklist

Operability Checklist For Production Jobs

Prove before, observe during, reconcile after.

⏱ 4 min · Topic 14 of 16

Before a job is production-ready, ask whether it can be tested, debugged, observed, and rerun.

The checklist is concrete: small tests, fixtures, golden outputs, mocked boundaries, row-count debug points, structured logs, health metrics, and a stored manifest.

If a failure happens, the job should leave a clear trail.

Core mental model

Prove before, observe during, reconcile after.

Why data engineers care

Production confidence comes from repeatable evidence, not hope.

Production readiness checklistworked example

Python

readiness = {
    "unit_tests": True,
    "golden_outputs": True,
    "structured_logs": True,
    "health_metrics": True,
    "stored_manifest": True,
}

Result · 5 rows

capability	required
unit_tests	true
golden_outputs	true
structured_logs	true
health_metrics	true
stored_manifest	true

Common mistake

Shipping a job that cannot explain its own row counts. Every incident becomes manual forensics.

Better habit

Review tests and observability together.

Make failure paths queryable.

Practice a rerun before the first incident.

Operator promise

A well-operated job leaves enough evidence that the next person can trust or reject its output quickly.

Remember this

Operability is the chapter where code becomes a service.

Testing, Debugging, Logging & Operability

What You'll Master Here

Testing Is Pipeline Evidence, Not Just Code Coverage

Unit Tests For Pure Transforms

Writing A Test For A Transform, Start To Finish

Fixture Design For Data Engineering

Golden Output Tests

Testing Rejected Rows And Failure Paths

Mocking Clocks, Files, APIs, And Databases

Debugging With Row Counts And Small Reproductions

Structured Logging With `logging.getLogger(name)`

Log Context: Batch ID, File, Partition, Row Counts

Metrics And Health Signals

Run Manifests, Audit Receipts, And Traceability

Operability Checklist For Production Jobs

Practice Lab

Next Chapter

Testing, Debugging, Logging & Operability

What You'll Master Here

Testing Is Pipeline Evidence, Not Just Code Coverage

Unit Tests For Pure Transforms

Writing A Test For A Transform, Start To Finish

Fixture Design For Data Engineering

Golden Output Tests

Testing Rejected Rows And Failure Paths

Mocking Clocks, Files, APIs, And Databases

Debugging With Row Counts And Small Reproductions

Structured Logging With `logging.getLogger(name)`

Log Context: Batch ID, File, Partition, Row Counts

Metrics And Health Signals

Run Manifests, Audit Receipts, And Traceability

Operability Checklist For Production Jobs

Practice Lab

Next Chapter