What You'll Master Here
Keep business logic pure, keep I/O at the edge, pass configuration explicitly, and make every boundary testable.
By now you can ingest files and turn raw strings into typed values. The next skill is structure: how to arrange that logic so a teammate can review it, test it, rerun it, and change it safely.
Data engineering Python usually fails when everything lives in one script: file reads, environment variables, parsing, validation, metrics, and writes all tangled together. This chapter separates those responsibilities.
You will learn to treat functions as pipeline boundaries, modules as ownership boundaries, and configuration as an explicit contract instead of a scattering of constants.
Keep business logic pure, keep I/O at the edge, pass configuration explicitly, and make every boundary testable.
Production jobs are changed more often than they are written. Clear functions and modules reduce review risk, isolate bugs, and make reruns explainable.
- pure function
- A function whose output depends only on its input and has no hidden side effects.
- I/O wrapper
- Thin code that reads, writes, logs, or calls external systems around pure logic.
- configuration object
- A single typed object that carries runtime settings through the job.
- entrypoint
- The file or function operators run to start the job, often guarded by __main__.
raw row
dict[str, object]
normalize_order()
typed transform
accepted
NormalizedOrder
rejected
RejectedRow
Putting parsing, environment reads, file writes, and transforms in one long function. A small rule change requires re-reading the whole job and makes unit tests awkward.
Write small functions around one decision.
Pass config as an argument instead of reading globals everywhere.
Keep run scripts thin and transform modules boring.
I would keep row transforms pure, isolate file/API/database I/O at the boundary, and pass a JobConfig object so the behavior is reviewable and testable.
Intermediate Python for data engineering is mostly boundary design: what belongs in the function, module, config, and entrypoint.