What You'll Master Here
Validate shape, validate fields, validate records, validate the batch, then write only safe output plus evidence.
Validation is how a Python data job proves that output rows are trustworthy before they reach a file, API, database, or warehouse.
This chapter turns earlier habits into a full contract system: required fields, field rules, record rules, batch rules, rejected rows, warnings, fatal errors, exception context, manifests, and tests.
The goal is not to make code noisy. The goal is to make failure paths explicit enough that bad data is stopped, explained, and measured.
Validate shape, validate fields, validate records, validate the batch, then write only safe output plus evidence.
Silent validation failures become bad metrics, unsafe upserts, broken partitions, and expensive incident investigations.
- rejected row
- A row excluded from accepted output with structured reason and source context.
- warning
- A non-blocking quality signal that is reported while the row may continue.
- fatal error
- A batch-level problem that should stop the run, such as missing required columns.
- contract evidence
- Counts and reports proving what validation accepted, rejected, warned, or stopped.
schema check
columns exist
field rules
types and ranges
record rules
row is coherent
batch rules
duplicates and totals
output
safe rows + evidence
Catching every error and continuing without evidence. The job appears resilient while silently losing correctness.
Separate warning, rejection, and fatal paths.
Attach file, line, field, value, and reason to failures.
Test each validation rule with tiny examples.
I would validate schema first, then field and record rules, emit rejected rows with source context, stop on fatal batch errors, and write a manifest that reconciles counts.
Validation is not just defensive code. It is the data contract made executable.