Serialization & File Formats

Inside Python, an order is a dict with a Decimal amount. Another program cannot read a Python dict. It reads a file.

Serialization means turning Python values into text or bytes for a file. Reading them back into Python values is called deserialization.

Every example uses the shop orders as a list of dicts:

from decimal import Decimal

orders = [
    {"order_id": 101, "customer_id": 1, "status": "paid",
     "amount": Decimal("40.00"), "order_date": "2026-01-03"},
    # ... orders 102 to 106 in the same shape ...
]

This chapter shows four file formats and when to use each:

A boundary is the place where data leaves your program. A contract is a promise about the shape of that data.

Some Python values have no text form that every reader understands. The json module refuses a Decimal:

import json
from datetime import datetime, timezone

row = {"order_id": 104, "amount": Decimal("15.50"),
       "paid_at": datetime(2026, 1, 15, 14, 30, tzinfo=timezone.utc)}

json.dumps(row)

So you decide the text form yourself, in one function, just before writing:

def to_wire(row):
    return {"order_id": row["order_id"],
            "amount": str(row["amount"]),
            "paid_at": row["paid_at"].isoformat()}

print(json.dumps(to_wire(row)))

Inside the program, keep Decimal and datetime. Turn them into text only at the boundary.

Chapter 4 read CSV files with the csv module. Writing uses the same module.

csv.DictWriter writes dicts as rows. fieldnames lists the columns, in order:

import csv

with open("orders_out.csv", "w", encoding="utf-8", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["order_id", "status", "amount"],
                            extrasaction="ignore")
    writer.writeheader()
    writer.writerows(orders[3:5])

print(open("orders_out.csv", encoding="utf-8").read())

Now read the file back:

with open("orders_out.csv", encoding="utf-8", newline="") as file:
    print(list(csv.DictReader(file)))

CSV stores no types. That is what schema-light means: the file has no description of its fields. The reader must know, from a separate agreement, that amount is a decimal number.

The csv module also adds quotes when a value holds a comma. Chapter 4 showed the reading side:

with open("notes.csv", "w", encoding="utf-8", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["order_id", "note"])
    writer.writerow([101, "Pen, Notebook"])

print(open("notes.csv", encoding="utf-8").read())

A JSON file holds one document, such as one big list. NDJSON means newline-delimited JSON: one JSON object on each line.

This helper turns one order into values JSON can write. Then each order goes on its own line:

def to_json_row(order):
    out = dict(order)
    out["amount"] = None if order["amount"] is None else str(order["amount"])
    return out

with open("orders.ndjson", "w", encoding="utf-8") as file:
    for order in orders[3:6]:
        file.write(json.dumps(to_json_row(order)) + "\n")

print(open("orders.ndjson", encoding="utf-8").read())

Unlike CSV, JSON keeps some types. 104 stays a number, and null stays a missing value.

Use NDJSON when you add records over time or read them one at a time. A new order is one new line. A broken line spoils only that line, as chapter 4 showed.

Use one JSON document for a single object, such as the settings of a run. A reader must load the whole document before it can use any part of it.

A round trip means writing a value and reading it back. You expect the same value. With JSON, some values come back different:

print(json.loads(json.dumps({1: "a", 2: "b"})))
print(json.loads(json.dumps({"items": (1, 2)})))

names = json.loads(json.dumps({101: "Asha"}))
print(names.get(101), names.get("101"))

JSON keys are always text, so the number key 101 came back as the text "101". A lookup by the number 101 now finds nothing. JSON also has no tuple, so a tuple comes back as a list.

Some values cannot be written at all, and one should not be:

from datetime import date

json.dumps({"day": date(2026, 1, 3)})

NaN means "not a number". Python can hold float("nan"), and json.dumps writes it without an error:

print(json.dumps({"v": float("nan")}))
print(json.dumps({"v": float("nan")}, allow_nan=False))

Two more settings make the output safe to compare. default=str turns any value json cannot write into text. sort_keys=True writes the keys in the same order every time:

print(json.dumps({"amount": Decimal("15.50")}, default=str))
print(json.dumps({"status": "paid", "amount": "40.00"}, sort_keys=True))
print(json.dumps({"amount": "40.00", "status": "paid"}, sort_keys=True))

Same text for the same data means a second run writes the same file. Then a quick compare shows if anything changed.

Compression makes a file smaller. It costs some computer time to write and read. gzip is a common compression for CSV and NDJSON files.

The gzip module opens a compressed file like a normal one. "wt" means write text:

import gzip
import os

lines = [json.dumps(to_json_row(o)) + "\n" for o in orders]
with open("orders.ndjson", "w", encoding="utf-8") as file:
    file.writelines(lines)
with gzip.open("orders.ndjson.gz", "wt", encoding="utf-8") as file:
    file.writelines(lines)

print(os.path.getsize("orders.ndjson"), os.path.getsize("orders.ndjson.gz"))

To read it, use gzip.open with "rt", which means read text. The lines come back exactly as written.

Parquet and Avro compress inside the file, so you do not gzip them.

CSV and NDJSON are row-oriented: they store one whole order, then the next. Parquet is columnar: it stores all order ids together, then all amounts together.

You can see the idea with Python lists. Here the same orders are stored by column:

columns = {
    "order_id": [o["order_id"] for o in orders],
    "amount": [o["amount"] for o in orders],
}

print(columns["order_id"])
print(sum(a for a in columns["amount"] if a is not None))

A report usually reads a few columns from many rows. A columnar file lets it read only those columns, so it reads much less.

A row format is better when you need whole records, one at a time. Examples are moving data between systems and writing events as they happen.

Data Pipeline chapter 20, Storage Layers & File Formats, shows this on the shop orders in more detail.

Parquet is a very common columnar file format for data lakes and warehouses. A Parquet file stores its schema: the name and type of each column.

So a reader gets numbers back as numbers, not text. It can also read only the columns it needs.

Python does not write Parquet by itself. You use a library such as pyarrow or pandas. This shows the shape of the code:

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.Table.from_pylist(orders)        # a list of dicts becomes a table
pq.write_table(table, "orders.parquet")

amounts = pq.read_table("orders.parquet", columns=["amount"])

ORC is another columnar format, common in older Hadoop and Hive systems. The idea is the same.

Avro is a row-oriented format. Each Avro file stores its schema, written as JSON, together with the rows.

Schema evolution means changing the schema over time, for example adding a field. Avro has rules, so old files and new readers can still work together.

Here is an Avro schema for orders. The last field is new:

{
  "type": "record",
  "name": "Order",
  "fields": [
    {"name": "order_id", "type": "int"},
    {"name": "status", "type": "string"},
    {"name": "coupon_code", "type": ["null", "string"], "default": null}
  ]
}

Some changes are safe and some break old data:

Use Avro when many producers and readers share one stream of records, and its fields change over time. In Python it needs a library such as fastavro.

A partition is a folder that holds the rows for one value, such as one date. The folder name says the value, for example order_date=2026-01-03.

A reader that needs one day opens one folder and skips the rest. Build the paths with pathlib (chapter 4):

from pathlib import Path

paths = sorted({str(Path("orders") / f"order_date={o['order_date']}" / "part-0.ndjson")
                for o in orders})
print(len(paths))
print(paths[0])

Six folders with one order each is a problem at scale. Many tiny files make readers slow. Partition by a value that keeps each folder big enough:

paths = sorted({str(Path("orders") / f"order_month={o['order_date'][:7]}" / "part-0.ndjson")
                for o in orders})
print(paths)

Pick the partition from how people read the data. If most reports ask for one day, partition by day. Avoid partitioning by a value with many different values, such as customer_id.

Start with the reader, not the writer. Ask who reads the file and how. Then this table gives a first choice:

Before you write a file, check five things:

1. Decimal, dates and None have a text form you chose on purpose.
2. CSV is written with the csv module, never by joining strings.
3. JSON keys are text when you read them back.
4. The same data gives the same bytes: sort_keys=True, no NaN.
5. The format and the partitions fit the way people read the data.

Check yourself: order 105 has amount None. What does a CSV file hold for it, and what does NDJSON hold? (Answer: an empty value, and null.)

Each exercise opens in the Python workspace with its own data. Write the code, run it, and check the result against the tests.

Each one is about what survives a write and a read, and what a second run must produce.