What You'll Master Here
values, field names, ordering, encoding, compression, schema, and file evidence.
Serialization is the moment Python values become bytes or text that another system must read. That boundary deserves the same care as validation or schema design.
This chapter covers CSV, JSON, NDJSON, compression, row-oriented formats, columnar formats, Parquet, Avro, ORC, partitioned layouts, and file manifests.
The standard-library examples are executable Python. The Parquet, Avro, and ORC sections explain the data engineering tradeoffs and where dedicated libraries or engines usually enter.
Serialization is an output contract: values, field names, ordering, encoding, compression, schema, and file evidence.
A perfectly normalized record can still fail downstream if Decimal, datetime, headers, compression, schema, or partition layout are serialized carelessly.
- serialization
- Converting in-memory values into a file, stream, or wire representation.
- row-oriented
- Records stored one row at a time, common in CSV, JSON, and NDJSON.
- columnar
- Values stored by column, useful for analytical scans and compression.
- manifest
- File-level evidence such as path, format, row count, schema version, and status.
domain values
Decimal / datetime
serializer
policy
bytes or text
file payload
manifest
evidence
Treating file writing as a final afterthought. Downstream jobs discover type and schema issues after the data lands.
Choose format by consumer and access pattern.
Serialize Decimal and datetime deliberately.
Write a manifest for every produced file.
I would pick the format based on consumer, schema needs, row vs column access, compression, and partition layout, then emit a manifest with row counts and schema version.
File format choice is data architecture, not a save-as detail.