Lakehouse Table Formats Theoretical

The reasoning

A folder of Parquet files has no notion of a table. A reader lists the directory and takes whatever is there. If a writer is halfway through, the reader sees half. If two jobs write at once, their files interleave. There is no schema except what the files happen to contain.

Delta Lake, Apache Iceberg and Apache Hudi each keep a metadata layer that says exactly which files are the table at version N. A commit adds a new version atomically. A reader picks one version and sees only its files, so a write in progress is invisible.

That metadata also carries the table's schema, its partition spec, and per-file statistics such as min and max per column. The engine prunes files from the metadata without listing storage, and evolves schema or partitioning by writing metadata, not by rewriting data.

The three formats differ in the shape of the metadata: Delta's ordered JSON log, Iceberg's tree of metadata files and manifests, Hudi's timeline. The guarantees are the same family.

The reasoning

Every write to a Delta table produces a new numbered JSON file: `000000000000000042.json`. It contains actions. `add` records each new data file with its statistics. `remove` records each file that is no longer part of the table. `metaData` and `protocol` record a schema or format-version change.

A reader reconstructs the table by replaying the actions in order: the set of files added and not removed is the table at that version. Because a commit is one file written atomically, a reader sees either the whole commit or none of it.

Replaying thousands of JSON files would be slow, so Delta writes a checkpoint: a Parquet file holding the full state at that version, every 10 commits by default. A reader loads the latest checkpoint and only the JSON files after it.

The log is also the history. `delta.logRetentionDuration`, documented at 30 days by default, decides how far back time travel can reach. The Delta docs say that each time a checkpoint is written, log entries older than the retention are cleaned up.

The reasoning

At the top, the catalog holds a pointer to the current `metadata.json`. That file holds the schema, the partition spec, the sort order, table properties, and the list of snapshots with their timestamps and parent ids.

A snapshot is the table at one commit. It points at a manifest list, an Avro file with one row per manifest, including the partition value ranges each manifest covers. A query filtering on `dt` can skip whole manifests from this level alone.

A manifest is an Avro file listing data files, and in format v2 also delete files, each with its partition, record count and column-level lower and upper bounds. The planner skips files whose bounds exclude the query's predicate.

A commit writes new manifests and a new manifest list, then a new metadata file, and finally swaps the catalog pointer. The Iceberg configuration page documents `write.metadata.previous-versions-max` at 100 and `commit.retry.num-retries` at 4.

The reasoning

Optimistic means nobody takes a lock while working. A writer starts from version 41, writes its new data files, and prepares version 42. The only critical section is the publish: a compare-and-swap of the table pointer from 41 to 42.

If a second writer already published 42, the swap fails. The writer then reads the new state and checks for a real conflict. Two appends to different partitions do not conflict, so it retries and publishes 43. Two updates to the same files do conflict, and one of them fails with a concurrent modification error.

Delta does this through the log directory, where writing commit file 42 must be atomic, which object stores need help with. Iceberg does it through the catalog's atomic pointer swap, with `commit.retry.num-retries`, documented at 4. Hudi's default is single writer; `OPTIMISTIC_CONCURRENCY_CONTROL` adds a lock provider for multi-writer.

Isolation is snapshot isolation for readers: a reader on version 41 keeps reading 41's files even after 42 lands, because the files are immutable and the metadata is versioned.

The reasoning

Parquet files are immutable, so a row cannot be changed in place. Under copy-on-write the engine finds the file that holds the row and reads the whole file. It writes a new file with the row changed. Then it records in the metadata that the old file is removed and the new one added.

The cost is proportional to the files touched, not the rows changed. Updating 1 row in a 120 MB file rewrites 120 MB. Updating 1,000 rows spread across 1,000 files rewrites 1,000 files. That is why copy-on-write suits batch updates that are rare or concentrated.

The reader's view is clean. Every file is the current truth, so a scan reads Parquet and nothing else, with full pushdown and statistics. BI and reporting tables, where reads dominate, favour copy-on-write.

It is the default in all three formats: Delta without deletion vectors, Iceberg's `write.update.mode` and `write.delete.mode` documented as copy-on-write, and Hudi's `hoodie.table.type` documented as `COPY_ON_WRITE`.

The reasoning

The write path is cheap: append a small file describing what changed and commit. A million small updates a day become a million small appends, not a million file rewrites. That is why merge-on-read suits streaming and CDC.

In Hudi, a merge-on-read table keeps a base Parquet file per file group and a stack of Avro log files holding updates and deletes for that group. In Iceberg format v2, a delete file marks rows to skip: a position delete names file and row number, an equality delete names column values. In Delta, a deletion vector is a bitmap of rows in a file that are logically deleted.

The read path pays. A snapshot query must read the base file, then apply the logs, delete files or vectors before returning rows. The more deltas accumulate, the slower and less predictable reads become, until compaction rewrites the base.

Hudi's `hoodie.table.type` set to `MERGE_ON_READ` chooses it per table. Iceberg chooses it per operation with `write.delete.mode`, `write.update.mode` and `write.merge.mode`, each documented as copy-on-write by default and available as merge-on-read on v2 tables.

The reasoning

Two problems create the need. Streaming writers commit small files every minute, so a table becomes millions of tiny files that are slow to list and open. Merge-on-read tables accumulate deltas, so every read merges more and more. Compaction pays both costs once, in the background.

Minor compaction merges the log or delete files among themselves without touching the base Parquet. It is quick and reduces the number of deltas a reader must apply. Major compaction reads the base and the deltas, merges by key, applies deletes, and writes a fresh base file. After it, a read is a plain Parquet scan again.

Triggers are configurable. Hudi's `hoodie.compact.inline.max.delta.commits` is documented at 5, with inline compaction off by default and async compaction on for Spark streaming. Delta's auto-compaction targets 128 MB files per its docs, and `OPTIMIZE` bin-packs toward larger ones. Iceberg's `rewrite_data_files` procedure targets `write.target-file-size-bytes`, documented at 512 MB.

Compaction is itself a write, so it commits optimistically and can conflict with other writers. The usual schedule is minor compaction often on bronze, major compaction on a schedule for silver, and no merge-on-read at all in gold.

The reasoning

A delete must make a row disappear from an immutable file. The copy-on-write answer is simple and expensive: find every file holding a matching row, rewrite each without it, commit the swap. Seven years of files for one user id can be tens of thousands of rewrites.

Iceberg v2's position delete is a small file saying "in `data-0042.parquet`, skip rows 17 and 9,331". It is precise and cheap to apply. An equality delete says "skip any row where `user_id = 42`", without knowing where those rows are. Flink CDC writers produce equality deletes because they cannot afford to look up positions per event; the price is that every read must scan for the values.

Delta's deletion vector is a compact bitmap stored beside the data file and referenced in the log. The file stays as it was; the reader masks the marked rows. `delta.enableDeletionVectors` turns it on per table.

All three leave the physical bytes in place until a rewrite. Proving a deletion for an auditor means compaction or `OPTIMIZE`, then `VACUUM` or snapshot expiry so no old version still references the file.

The reasoning

The statistics are collected when a file is written. Delta stores them in the `add` action of the log. Iceberg stores lower and upper bounds per column in the manifest entry. Hudi keeps them in its metadata table's column stats index. None of this needs the file to be opened at query time.

Skipping works best when values are clustered. If `order_id` increases with time and files are written in order, each file covers a narrow range and a point lookup touches one file. If files hold random ids, every file's range covers the whole space and nothing is skipped.

That is why sorting or Z-ordering is a read optimisation: it narrows the ranges. Partitioning is the coarse version of the same idea, skipping whole folders or manifests by the partition value.

Delta's docs note that statistics are collected only for the first columns, up to `delta.dataSkippingNumIndexedCols`, and each nested field counts separately. A wide table may need its filter columns moved forward, or the property raised.

The reasoning

Sorting by one column narrows that column's range per file and leaves the others random. Z-ordering interleaves the bits of several columns into one sort key, so files are compact in all of them, at some cost to each. Delta's docs say effectiveness drops with every extra column, and to use it on high-cardinality columns that appear in predicates.

Z-order is a full rewrite. It is not idempotent, and new appends are not Z-ordered until the next `OPTIMIZE`. On a table that grows every hour, the table is always partly unordered.

Liquid clustering, `CLUSTER BY` on a Delta table, chooses the clustering columns at creation and clusters incrementally as `OPTIMIZE` runs, touching only new or poorly clustered files. The columns can be changed later without rewriting data, which partitioning cannot do.

Partitioning remains right for very low-cardinality columns that every query filters on, such as a date, and for isolating data that is deleted or retained together. For everything else, clustering gives the skipping without the small-file problem partitions create.

The reasoning

Every file costs a listing entry, an open, a footer read, a metadata row and a task. A terabyte in 2 million 500 KB files spends most of its time on that overhead. The same terabyte in 2,000 files of 512 MB reads at line speed.

Files can also be too large. A 10 GB file is one task for a reader that cannot split it, and under copy-on-write a 1-row update rewrites all 10 GB. Row groups inside Parquet, documented at 128 MB in Iceberg's `write.parquet.row-group-size-bytes`, give readers a unit smaller than the file.

The writer controls size at write time through the engine's partition count, and after the fact through compaction toward the target. Streaming writers always produce small files and rely on compaction.

The right size depends on the engine's split size and the query pattern. Scan-heavy tables lean larger; tables with frequent point updates under copy-on-write lean smaller to limit rewrite cost.

The reasoning

A Hive table partitioned by date needs an explicit `event_date` column that writers must fill and readers must filter on. Filter on `event_time` instead and nothing is pruned. Iceberg replaces the column with a transform in the spec: `day(event_time)`, `hour`, `month`, `bucket(16, user_id)`, `truncate`.

Because the spec is metadata, Iceberg derives the partition value at write time and translates a predicate on the source column into a predicate on the partition at read time. Users never see or maintain a partition column.

Partition evolution is a metadata change. Switching a table from `day(event_time)` to `hour(event_time)` adds a new spec id. New writes use it. The three years of old files keep spec 0, and a query plans across both, pruning each manifest by its own spec. Nothing is rewritten.

Hive-style tables cannot do this: changing the partition column means rewriting every file into new directories. Delta's answer to the same need is liquid clustering, where the clustering columns can change without a rewrite.

The reasoning

Delta's docs give the syntax: `SELECT * FROM t VERSION AS OF 41` or `TIMESTAMP AS OF '2026-10-01'`, and `.option('versionAsOf', 41)` on a DataFrame read. Iceberg exposes snapshots the same way, with `FOR VERSION AS OF` and snapshot ids. Hudi reads a table as of an instant on its timeline.

The mechanism is a different starting point in the metadata. Version 41's log entries, or snapshot 41's manifest list, name the files that were the table then. Those files were never modified, only superseded, so reading them gives the exact old state.

Two things cut it off. Log retention: Delta's `delta.logRetentionDuration` is documented at 30 days, so older versions are not reconstructable. File retention: `VACUUM` in Delta and `expire_snapshots` in Iceberg remove data files no current version references, after which an old version points at files that are gone.

Time travel is also the basis of rollback and of auditing a bad write. Diff the current version against the last known good one, then restore it as a new commit.

The reasoning

When a commit removes a file from the table, the bytes stay on storage. That is what makes time travel and concurrent readers work. It is also why a table with hourly merges grows to many times its live size unless something deletes the old files.

`VACUUM` walks the table directory and deletes files that are no longer referenced and were removed before the retention threshold. The Delta docs say data files are deleted only by `VACUUM` and that it does not delete log files. Running it with a short retention is dangerous: a job that started reading version 40 an hour ago still needs 40's files.

Iceberg separates the two steps. `expire_snapshots` removes old snapshots from the metadata, keeping at least `history.expire.min-snapshots-to-keep`, documented at 1, then deletes the data and manifest files only those snapshots referenced. Metadata files themselves are capped by `write.metadata.previous-versions-max`, documented at 100.

Orphan files are a separate problem: files written by a job that failed before committing, which no snapshot ever referenced. Neither operation above finds them, because they are not in any metadata. Iceberg's `remove_orphan_files` and a periodic listing against the metadata are the cleanup.

The reasoning

A full scan reads the whole table every run. An incremental read asks "what changed since version 41" and gets inserts, updates and deletes with a marker on each. The downstream job applies those to its own table with a merge, at a cost proportional to the change, not the table.

Delta writes change files into a `_change_data` folder on commits that update or delete rows when the feed is enabled, and derives inserts from the added files. A reader uses `readChangeFeed` with a starting version or timestamp and gets `_change_type`, `_commit_version` and `_commit_timestamp` columns.

Iceberg's `create_changelog_view` procedure builds a view of net changes between two snapshots, and incremental append scans read only files added in a snapshot range. Hudi's incremental query mode reads rows with a commit time after the one you pass, which is how a 15-minute `fact_trips` build reads only new trips.

The feed is only as retained as the history behind it. Once `VACUUM` or snapshot expiry removes the versions, the changes are gone and the consumer must fall back to a full read.

The reasoning

Old files are immutable, so a schema change can only be a rule for interpreting them. Adding a column means old files return null for it. Widening int to long means old int values are read as long. Those are safe because every old file can still be read under the new schema.

Dropping a required column breaks readers that expect it. Narrowing a type can lose values. Changing a string to an int fails on old files that hold non-numeric strings. Those need a migration: a new column, a backfill, a cutover.

Name-based formats have a trap. Drop `discount`, later add a new `discount` with a different meaning, and old files' `discount` values flow into the new column. Iceberg assigns each column an id at creation and stores ids in the file metadata. The new `discount` has a new id, so old files correctly return null for it.

Delta enforces the schema on write and rejects a mismatched append unless `mergeSchema` or `overwriteSchema` is set, and uses column mapping to allow renames and drops. Delta's docs call session-wide automatic merging not recommended.

The reasoning

On a copy-on-write table, snapshot and read-optimized queries are the same thing, because there are no log files. The difference only appears on merge-on-read.

A snapshot query on a merge-on-read table reads each file group's base file and applies its log files. It sees every committed update, at the cost of a merge per read. Hudi's docs document `hoodie.datasource.query.type` with `snapshot` as the default.

A read-optimized query reads base files only. It is as fast as a Parquet scan and it does not see updates that are still only in logs, so it shows the table as of the last compaction. Dashboards that can accept minutes of staleness use it.

An incremental query takes a begin instant and returns rows that changed after it, using the timeline to find the commits and the files they touched. It is how downstream tables are fed cheaply.

The reasoning

Hudi needs to know where a record lives before it can update it. Without an index, an upsert would have to scan the whole table to find matching keys. The index answers "which file group holds `order_id = 42`" from metadata.

Index types trade lookup cost against write cost. Bloom uses bloom filters and key ranges stored in file footers; it suits keys that are mostly new and roughly ordered. Simple joins incoming keys against the table's keys; Hudi's docs name it the default on Spark. Bucket hashes keys to a fixed set of file groups with no lookup at all. The record-level index in the metadata table gives a direct key-to-file map.

During an upsert, the batch is first deduplicated by key using the ordering field. Each key is looked up in the index. The batch is then split: updates go to their existing file groups, and inserts are packed into new or small file groups. Then each affected group is written, as a rewrite under copy-on-write or as a log file under merge-on-read.

Random keys such as UUIDs defeat bloom and range pruning, because no file can be ruled out, so lookups grow with table size. That is the usual reason an upsert that took 3 minutes at 500 million rows takes 40 at 5 billion.

The reasoning

A table format's files describe the table, but something must say where the current metadata is. For Delta that is the `_delta_log` directory itself, so Delta needs a catalog mainly for names and permissions. For Iceberg the catalog is essential: it holds the pointer to the current `metadata.json` and the compare-and-swap that commits a new one.

The Hive metastore was built for Hive tables and stores Iceberg's pointer as a property. It works, but it is a database every engine must reach directly. AWS Glue plays the same role as a managed service in one cloud.

The Iceberg REST catalog specification turns the catalog into an API. Engines call it to load tables, commit, and increasingly to get credentials, so access control sits in one place and no engine needs database credentials. Nessie and Polaris are implementations, and Unity Catalog and Glue now speak it too.

The catalog is also where multi-engine interoperability lives or dies: Spark, Trino, Flink and a warehouse can share a table only if they agree on the catalog that names it.

The reasoning

A `MERGE INTO target USING source ON key` first scans the target for files that might contain source keys, using statistics to skip the rest. Only the touched files are read into memory.

For each touched file, the engine rewrites it with matched rows updated or removed and, in the same job, writes new files for the unmatched source rows. Nothing is edited in place; the result is a set of new files and a list of old files to retire.

The commit records removed and added files atomically. A reader on the old version keeps its files; a reader after the commit sees the new ones. If the job fails before commit, the new files are orphans and the table is untouched.

The cost drivers are the number of files touched and their size. Clustering the target on the merge key, limiting the merge to recent partitions, and keeping files at the target size all make merges fast. High-frequency small merges favour merge-on-read instead.

The reasoning

The lake gave cheap, open, unlimited storage and no reliability: no transactions, no schema, no safe updates. The warehouse gave reliability and performance in a proprietary engine that owns the data. Teams ended up copying data between the two.

A table format on the lake closes most of the gap. Delta, Iceberg and Hudi add a metadata layer that gives atomic commits, snapshot isolation, schema enforcement and statistics. The lake behaves like a table store while the files stay open Parquet.

That lets one copy of the data serve many engines: Spark for ETL, Trino or a warehouse for BI, Flink for streaming, Python for ML. The warehouse becomes one reader among several instead of the owner.

What a lakehouse still lacks relative to a warehouse is the integrated engine: compaction, statistics and clustering are jobs you run, not things that happen.

The reasoning

Bronze takes a stream of small commits: events every minute, CDC changes every few seconds. Rewriting base files for each would be prohibitive. Merge-on-read makes those writes cheap, and bronze readers are batch jobs that can afford a merge.

Silver tables fed by CDC with many updates are the same case. If silver is rebuilt in batch from bronze once an hour, copy-on-write is fine, because the write is one large rewrite anyway.

Gold is read thousands of times by BI tools and updated rarely, in scheduled batches. Copy-on-write gives those readers plain Parquet with full statistics and no merge, and the occasional rewrite is a price paid once a day.

Whatever the layer, a merge-on-read table needs compaction scheduled against its read SLA. A copy-on-write table needs its merges aligned to partitions or clustering, so each update touches few files.

The reasoning

Row groups make a file splittable: a reader can process row group 3 without the others, which is how one large file becomes several tasks. Iceberg documents `write.parquet.row-group-size-bytes` at 128 MB.

Within a row group, a column chunk holds all values of one column, divided into pages. Because the values are one type and often similar, each chunk picks an encoding that fits: dictionary for repeated strings, run-length for sorted or clustered values, delta for increasing numbers. A codec such as Snappy or Zstd compresses on top.

The footer is read first. It carries the schema, the row group boundaries, and for each column chunk the min, max and null count. A query for `region = 'EU'` skips any row group whose region range excludes it. A query for 3 of 40 columns reads only 3 chunks per row group.

Parquet files are immutable and written in batches, which is why streaming writers produce many small files and why updates need a table format above the file.

The reasoning

Atomicity on a file system can lean on rename. Object stores do not rename, so Delta makes the commit the creation of one log file, which must be atomic and exclusive. On S3 that needs a coordination service or a conditional put. Iceberg makes the commit the catalog's pointer swap, which is atomic in the catalog's own store.

Consistency is schema enforcement and invariants: a write that violates the schema or a constraint is rejected, so no version is ever malformed.

Isolation is snapshot isolation. A reader fixes a version at the start and reads only its files. Writers work optimistically and check for conflicts when they publish, so two jobs never corrupt each other, though one may fail.

Durability is inherited from the object store's durability once the commit is written. Before the commit, the new files are just bytes nobody references.

The reasoning

The log is a list of JSON commits, and the table state is the result of replaying them. After thousands of commits, replaying from zero would take longer than the query. The checkpoint is the saved result of that replay.

A reader finds the latest checkpoint through the `_last_checkpoint` file, loads it, then applies the commits with higher version numbers. The Delta docs note that each time a checkpoint is written, Delta cleans up log entries older than the retention interval.

Checkpoints are also where the cost of a huge table shows: a table with millions of files has a large checkpoint, which is why file count matters even for metadata. Newer Delta versions support multi-part and v2 checkpoints to keep this manageable.

Checkpoints never contain data rows. They are metadata about files, the same actions as the JSON commits in a columnar form.

The reasoning

Compaction is a write like any other: new large files are written, the old small files are marked removed in the log, and readers after the commit see fewer files. The old files stay until `VACUUM`.

Delta's docs say bin-packing is idempotent, so running `OPTIMIZE` on an already compacted table is a no-op. With `ZORDER BY`, the docs say it is not idempotent, because the whole clustering is recomputed and files are rewritten again.

`OPTIMIZE` can be scoped with a `WHERE` on partition columns, so a nightly job compacts only yesterday's partition rather than the whole table. Auto-compaction and optimized writes do a lighter version at write time, targeting 128 MB per the docs.

Because it is a transaction, `OPTIMIZE` can conflict with concurrent writers on the same files. Scheduling it against quiet partitions avoids retries.

The reasoning

The guarantee rests on immutability. A commit never edits a file; it writes new files and a new metadata version. A reader holding version 41's file list is unaffected by version 42, because 41's files still exist unchanged.

That is also why vacuum has a retention window. If a cleanup deleted files the moment they were superseded, a running query on version 41 would fail mid-scan when its files vanished.

Writers get the same isolation among themselves: each works from the version it read, and the commit step detects whether another writer changed the files it depends on.

The practical limit is that a query sees the table as of its start, not its end. A dashboard refreshed during a load shows the previous load, which is correct, and the next refresh shows the new one.

The reasoning

The data files are the same Parquet in all three formats. Only the metadata differs. So a translator can read Delta's log and write an Iceberg manifest tree pointing at the same files, and an Iceberg-only engine can then read the table.

UniForm is built into Delta: enable it on a table and each Delta commit also produces Iceberg metadata, which a catalog can expose. It is one-directional, Delta as the source of truth, and the Iceberg view may lag a commit.

XTable, formerly OneTable, runs outside any engine. It syncs metadata in any direction between the three formats, incrementally, on a schedule or after a job. It is the choice when the writer cannot be changed or when the source of truth is not Delta.

Neither makes the table writable from the second format safely. They are read-side bridges, and the translated metadata has its own retention and sync lag to manage.

The reasoning

Version 1 defined the metadata tree, snapshots, hidden partitioning and schema evolution by column id. Every write produced whole new data files; a one-row delete meant rewriting the file.

Version 2 introduced delete files as first-class members of a snapshot. A manifest can list delete files alongside data files, and a reader applies them. That is what makes `write.delete.mode`, `write.update.mode` and `write.merge.mode` settable to merge-on-read, which the docs restrict to v2 and above.

Upgrading a table from v1 to v2 is a metadata change. Existing files are untouched; new writes may use delete files. Readers must be on an engine version that understands v2, which is why the version is a reserved table property rather than something a writer sets freely.

Version 3 continues the line with row lineage, deletion vectors and new types, which is where the formats are converging.

The reasoning

Min and max statistics prune ranges, but a point lookup on a high-cardinality column inside a wide range still has to open the file. A bloom filter answers membership instead: hash the value into a few positions and check whether every bit is set.

Parquet can store a bloom filter per column chunk, which readers consult after the footer statistics. Hudi's bloom index stores filters in file footers and uses them, with key ranges, to decide which file groups an upsert must probe.

The trade is size against false-positive rate. More bits per key means fewer false positives; the documentation for most implementations targets around 1 percent at about 10 bits per key. A false positive costs one unnecessary file read, never a wrong answer.

Bloom filters help when keys are selective and files are many. On random UUID keys they still prune, which is why Hudi reaches for them. But their size grows with the number of keys, and they cannot answer range predicates at all.