What You'll Master Here
it records which files make up the table at each version, giving ACID, time travel, and safe concurrent writes.
Plain Parquet on a data lake is fast and cheap, but it has no transactions: a failed write leaves half-written files, two writers can corrupt each other, and there is no schema enforcement or update/delete. Delta Lake adds a transaction layer on top of Parquet that fixes all of this, turning the lake into a lakehouse: data-lake economics with data-warehouse reliability.
This chapter covers what Delta adds, ACID transactions (via a transaction log), time travel (query past versions), schema enforcement and evolution, MERGE for upserts, and the maintenance operations (OPTIMIZE, Z-order, VACUUM) plus Auto Loader for incremental ingestion. It is the storage foundation of modern Spark platforms.
With a diagram of the Delta transaction log and a worked MERGE/upsert example. The big picture: Delta is why you can run reliable, updatable tables directly on cheap object storage.
Delta = Parquet files + a transaction log. The log is the source of truth: it records which files make up the table at each version, giving ACID, time travel, and safe concurrent writes.
The lakehouse (Delta/Iceberg/Hudi) is the dominant modern data architecture. Delta Lake skills, especially MERGE, time travel, and OPTIMIZE, are expected of any Spark engineer and feature heavily in interviews.
- lakehouse
- Lake storage economics plus warehouse reliability (ACID, schema, updates).
- transaction log
- Delta's ordered record of commits defining the table at each version (_delta_log).
- ACID
- Atomic, consistent, isolated, durable transactions on the lake.
- time travel
- Querying a previous version (or timestamp) of a Delta table.
Treating a Delta table as just a folder of Parquet files. Editing files directly bypasses the transaction log and corrupts the table, always go through Delta operations.
Use Delta (not raw Parquet) for tables that are updated or written concurrently.
Use MERGE for upserts/CDC instead of overwrite hacks.
Run OPTIMIZE/VACUUM as routine maintenance.
Delta wraps Parquet with a transaction log. That log is what delivers ACID, time travel, schema enforcement, and safe upserts, the reliability of a warehouse on the cost and scale of a lake.
Delta Lake adds a transaction log over Parquet to give ACID transactions, time travel, schema enforcement/evolution, and upserts (MERGE), the lakehouse. It is the reliable, updatable storage foundation for modern Spark platforms.
- What does Delta add on top of plain Parquet, and how?
- Why is the transaction log central to Delta's guarantees?