Data Transforms · Knowledge Base
Python for data engineering, foundations to production. Parsing and cleaning messy inputs, transformation patterns that stay readable, testing and validation, and the habits that survive real pipelines.
StartThink in records, transforms, contracts, failure paths, and reviewable code instead of isolated syntax drills.
Read chapter →Lists, dicts, sets, tuples, grouping, lookup maps, dedupe keys, and nested records used in real transforms.
Read chapter →Clean, parse, and normalize messy text: encodings, whitespace, casing, delimiters, and regex for real-world fields.
Read chapter →Read files and payloads safely with pathlib, CSV, JSON, NDJSON, encodings, manifests, and malformed-row handling.
Read chapter →Handle None, datetime values, time zones, Decimal math, rounding, optional fields, and type hints deliberately.
Read chapter →Structure transform code with small functions, modules, config objects, environment rules, and dependency boundaries.
Read chapter →Process data that does not fit in memory using lazy evaluation, yield, itertools, and streaming file pipelines.
Read chapter →Use map/filter/reduce-style flows, in-memory joins, aggregations, stateful scans, and sorted-window logic.
Read chapter →Move data between CSV, JSON, Parquet, Avro, and ORC: row vs columnar, compression, and schema-aware reads/writes.
Read chapter →Design required-field checks, schema validation, rejected-row reports, warnings, exceptions, and contract evidence.
Read chapter →Pull from APIs and databases with pagination, retries, rate limits, secrets boundaries, and idempotent source reads.
Read chapter →Choose threads, processes, or asyncio with the GIL in mind, then profile and tune before reaching for a cluster.
Read chapter →Use pytest-style thinking, fixtures, golden outputs, logging, metrics, and traceable failures for production confidence.
Read chapter →Build cursor, watermark, replay-window, checkpoint, manifest, and task-boundary habits for rerunnable jobs.
Read chapter →Know when to move from pure Python to pandas, Polars, or Spark, and how to narrate those tradeoffs in interviews.
Read chapter →