Lakehouse Table Formats Conceptual

The reasoning

A folder of Parquet is not a table, it is a naming convention. The "table" is whatever a reader finds when it lists the directory, so its contents change **file by file** as a job writes. A reader that lists mid-job sees some of the new files and not others; a job that dies half way leaves partial output that every later reader treats as real data. There is no commit, so there is nothing to roll back to.

Two writers make it worse. Each writes files and, for overwrites, deletes the old ones; neither knows about the other. Two `INSERT OVERWRITE` jobs on the same partition can interleave so the result holds files from both, or from neither. Nothing detects it, because nothing records what the table was supposed to contain.

A table format fixes this by making **the list of files part of the table**. Delta keeps an ordered log of JSON commits in `_delta_log`; Iceberg keeps a tree of metadata file → manifest list → manifests → data files, with the current metadata file pointed to by a catalog; Hudi keeps a timeline of instants under `.hoodie/timeline`. In all three a write puts its data files down first and then publishes them in one atomic step. Readers only ever see committed versions, and a failed job leaves files nobody references.

Everything else follows from that list. Row-level `UPDATE`, `DELETE` and `MERGE` become possible because a commit can say "these files are replaced by those". Time travel is reading an older list. Schema and partition changes become metadata edits. Query planning can use file statistics stored in the metadata instead of listing directories. The honest caveat: the files are still immutable Parquet, so every one of those features is paid for in rewrites, extra files or maintenance jobs.

The reasoning

**Where Hive's cost comes from.** A Hive table is two systems glued together: the metastore knows the partitions, and the file system knows the files. To plan a query the engine asks the metastore which partitions match, then **lists each partition's directory** to find the files. On object storage a listing is a paged API call, so 40,000 matching partitions means tens of thousands of requests before a single byte of data is read. The listing also reflects whatever is there at that instant, including files from a job that is still writing.

**Iceberg's tree.** The catalog stores one pointer: the location of the table's current `metadata.json`. That file holds the schema, partition specs and the list of snapshots. Each snapshot points to a **manifest list**, which lists manifests together with summary information such as partition value ranges and file counts. Each **manifest** lists data files (and delete files) with their partition values and column-level stats such as lower and upper bounds and null counts. The Iceberg reliability docs put it as O(1) RPCs to plan, instead of listing O(n) directories.

**Pruning happens in the metadata.** For `WHERE event_date = '2026-03-01'` the planner skips whole manifests whose partition range excludes that date, then skips data files whose stats exclude it, and only then opens Parquet. Because the file list is explicit, a half-written job is invisible: its files are not in any committed manifest.

**What it does not fix on its own.** Iceberg removes the listing cost, not the small-file cost. A table with millions of 2 MB files still has millions of manifest entries and millions of file opens; that is what `rewrite_data_files` and `rewrite_manifests` are for.

The reasoning

A **snapshot** is a table version. It has an id, a parent id, a sequence number, a timestamp, a summary (operation, added and deleted file counts) and one field that matters most: the location of its **manifest list**. The manifest list is the snapshot's index of manifests; the manifests are the index of files.

For an append of 10 files to a 5,000-file table the writer: (1) writes the 10 Parquet files; (2) writes **one new manifest** listing those 10 files with their partition values and column stats; (3) writes a **new manifest list** containing the new manifest plus references to the existing manifests, which it does not rewrite; (4) writes a new `metadata.json` that adds the snapshot and makes it current; (5) asks the catalog to swap the pointer from the old metadata file to the new one. Nothing about the other 5,000 files is touched.

That reuse is the point. The spec describes each write as producing a new snapshot that reuses as much of the previous metadata tree as possible, and it assigns every commit a **sequence number** that new files inherit. Sequence numbers are how Iceberg later knows which delete files apply to which data files: an equality delete only applies to data files with a lower sequence number.

The cost shows up elsewhere. Each append adds a manifest, so a table written every minute accumulates thousands of small manifests; each commit adds a metadata file, and old ones are only removed if `write.metadata.delete-after-commit.enabled` is set (default `false`, keeping up to `write.metadata.previous-versions-max`, default 100, tracked). `rewrite_manifests` regroups manifests; `expire_snapshots` drops old snapshots and the files only they reference.

The reasoning

Every change to a Hudi table is an **instant** on the timeline. An instant has an action type, a requested time that acts as the transaction id, a completion time, and a state: `REQUESTED`, `INFLIGHT` or `COMPLETED`. The action types in the 1.2 docs include `COMMIT` (copy-on-write writes), `DELTA_COMMIT` (merge-on-read writes), `REPLACE_COMMIT` (insert-overwrite and clustering), `COMPACTION`, `LOGCOMPACTION`, `CLEAN`, `ROLLBACK`, `SAVEPOINT`, `RESTORE` and `INDEXING`.

**Atomicity comes from the state transition.** A writer creates the requested instant, writes its data files, and then marks the instant completed. Until that last step, readers and query engines ignore the files it wrote, because file slices are only visible if the instant that produced them completed. A job that crashes leaves an inflight instant; the next writer or the table services roll it back and delete its files.

**Ordering and isolation.** Hudi 1.x orders actions by completion time with what the docs call TrueTime semantics, which gives readers snapshot isolation and lets table services (compaction, clustering, cleaning) run alongside writers under MVCC. Multiple writers need a concurrency mode and a lock provider — the default is `SINGLE_WRITER`.

**Hudi 1.0 changed the storage of history.** Before 1.0 there was an active timeline and an archived one; 1.0 replaced the archive with an LSM-tree based history so long histories stay readable. That matters because incremental and time-travel queries read the timeline to find which files changed between two instants.

The reasoning

**Record key** (`hoodie.datasource.write.recordkey.field`) is the row's identity — `order_id`, or a composite via `ComplexKeyGenerator`. Hudi uses it to find existing versions on upsert and delete. In 1.x the key is optional for pure `INSERT` and `BULK_INSERT` loads: if none is configured Hudi generates keys, but then there is nothing to upsert against.

**Partition path** (`hoodie.datasource.write.partitionpath.field`) decides the directory. It interacts with the key: with a non-global index uniqueness is only enforced **within** a partition, so if an order's partition is `order_status` and the status changes, the new version lands in another partition and you now have two live copies. Partition on something immutable for the record's life (creation date), or use a global index.

**Ordering field** — the precombine field in 0.x, `hoodie.table.ordering.fields` in 1.x — resolves conflicts. With one set, the merge mode defaults to `EVENT_TIME_ORDERING` and the version with the highest ordering value (say `updated_at`) wins, both within an incoming batch and against what is stored. With none set, it defaults to `COMMIT_TIME_ORDERING`: the latest write wins even if it carries older data.

**Key to file group.** Within a partition Hudi stores data in **file groups**, each with a stable file id. The index maps key (plus partition, for non-global indexes) to a file id, and the docs are explicit that this mapping never changes once the first version is written. So `order_id = 42` always goes to the same file group; an update to it produces a new file slice there (copy-on-write) or appends a log block to it (merge-on-read).

The reasoning

**Copy-on-write (COW).** The docs define it as: updates or deletes create new base files in the file group, and no log files are written. To change one row the writer finds its file group, reads the current base file, writes a complete new version with the row changed and commits it as a new file slice. Write amplification is the whole file for one row; read cost is plain Parquet.

**Merge-on-read (MOR).** Updates go to row-based **log files** attached to the file group; the base file is untouched. The write is small and fast, so MOR suits frequent upserts. A **snapshot query** has to read the base file and merge the log records on the fly, so read latency rises as logs accumulate. A **read-optimised query** reads only base files — fast, but stale by whatever is sitting in the logs.

**Compaction** closes the loop on MOR: it merges logs into a new base file, producing a new file slice. You trade when you pay: COW pays at write time, on every write; MOR pays at read time until compaction, and then pays the rewrite once for many updates. The table type is fixed at creation (`hoodie.table.type` in the table config, set on first write with `hoodie.datasource.write.table.type`).

The rule of thumb that follows from the mechanism, not a benchmark: COW when the table is read far more than it is updated and updates arrive in large batches; MOR when updates are frequent and spread thinly across many files, such as CDC every few minutes, and you can afford to run compaction.

The reasoning

Format v2 added **delete files** so rows can be removed without rewriting data files. A **position delete** says "row 1,337 of file `a.parquet` is deleted". An **equality delete** says "any row where `order_id = 42` is deleted", and applies to data files with a lower sequence number in the same partition. Both are stored as files and tracked in manifests like data files.

**Which writer produces which.** Spark `DELETE`, `UPDATE` and `MERGE` with `write.delete.mode`, `write.update.mode` or `write.merge.mode` set to `merge-on-read` first scan the table to find matching rows, so they know file and position and write position deletes. A streaming upsert writer like Flink applying CDC with identifier fields does not want to scan the table per event, so it writes an equality delete for the key plus the new row. Cheap to write, expensive to read: every reader has to join equality deletes against older data files.

**Format v3 changes position deletes.** The v3 spec replaces position delete files with **deletion vectors** — a bitmap per data file stored in Puffin files — and says writers must not add new position delete files to v3 tables (existing ones stay valid). Equality deletes are unchanged. Which version a new table gets depends on the engine and the `format-version` table property it was created with, so check which a table is before assuming either.

**Why it matters operationally.** Delete files accumulate. `rewrite_data_files` with a `delete-file-threshold` rewrites data files with their deletes applied; `rewrite_position_delete_files` compacts position deletes. Without them, read cost grows with every CDC commit.

The reasoning

**1. Pre-combine.** Within the incoming batch, rows with the same record key are reduced to one using the ordering field (`EVENT_TIME_ORDERING` keeps the highest `updated_at`). Skipping this with a batch that contains two versions of a key is how duplicates or stale versions get in.

**2. Index lookup (tagging).** Each key is looked up in the index to find the file group that holds it. This is the expensive step on large tables and it is why the index type matters: a bloom index checks bloom filters and key ranges per file, a simple index joins against keys read from storage, a bucket index hashes the key to a bucket with no lookup at all, and the record-level index looks keys up in the metadata table.

**3. Partition the work.** Tagged rows are updates to known file groups; untagged rows are inserts. Hudi's file sizing assigns inserts to existing small files in the partition, up to the configured maximum, before creating new file groups, so upserts do not create a new small file every commit.

**4. Write and commit.** For copy-on-write, each touched file group gets a new base file; for merge-on-read, updates are appended to log files. The write runs under an inflight instant; when it completes, the instant flips to `COMPLETED` and readers see the new file slices. Cleaning later removes the file slices the new versions superseded, keeping `hoodie.clean.commits.retained` commits (default 10) for readers still using them.

The reasoning

The 1.2 docs describe the index as a mapping from record key (plus partition path, for non-global indexes) to a file id. The choice is `hoodie.index.type`; the default is `SIMPLE` for Spark and Java, and in-memory or Flink state for Flink.

**BLOOM** checks each file's bloom filter and min/max key range. It prunes well when keys are ordered — time-prefixed ids where updates hit recent files — and poorly when updates scatter across the whole table, because every file's filter says "maybe". **SIMPLE** reads keys from the files in affected partitions and joins them with the incoming keys: predictable, and a good fit for random updates in modest partitions.

**BUCKET** hashes the key into a fixed number of buckets, each a file group, so there is no lookup at all. With the `SIMPLE` engine the bucket count is fixed per partition; `CONSISTENT_HASHING` resizes buckets but is merge-on-read only. Bucket indexes are also what Hudi's non-blocking concurrency control requires. **RECORD_LEVEL_INDEX** (partitioned, added in 1.1) and **GLOBAL_RECORD_LEVEL_INDEX** store an exact key-to-location map in the metadata table, which suits very large tables where bloom and simple lookups have become the bottleneck. `RECORD_INDEX` is deprecated in favour of those two.

**Global vs non-global** is a separate choice: global indexes (`GLOBAL_BLOOM`, `GLOBAL_SIMPLE`, the global record-level index) enforce one live record per key across all partitions, at the cost of looking in every partition; non-global indexes only guarantee uniqueness within one. Hudi 1.x also adds secondary and expression indexes in the metadata table, which speed up reads on non-key columns rather than upserts.

The reasoning

An upsert on immutable files is always "read the affected files, write new ones, retire the old ones". What makes it safe is the last step being **atomic** and **validated**. Delta commits a JSON log entry that removes old files and adds new ones; Iceberg commits a new snapshot through a pointer swap and checks that the files it replaced still exist; Hudi completes an instant on its timeline. Readers see the old version or the new one, never a mix.

Plain Hive on Parquet has none of that. The usual workaround is `INSERT OVERWRITE` of every partition that contains a changed row: read the partition, join with the changes, write the whole partition back. Between deleting the old files and finishing the new ones, a reader sees a partial partition. If two jobs do it at once, the last one to finish wins and the other's changes are gone. (Hive ACID tables in ORC do support MERGE with their own delta files and compaction, but that is a Hive-specific transactional table, not a Parquet folder any engine can read.)

The formats also narrow what a merge rewrites. Iceberg's Spark docs recommend `MERGE INTO` over `INSERT OVERWRITE` because it can replace only the affected data files rather than whole partitions. Hudi's index finds the file groups for the incoming keys. With merge-on-read in either format, the merge writes delete files or log files instead of rewriting data at all.

So the precise answer is: Delta Lake, Hudi and Iceberg (format v2 and above for row-level merge-on-read) all support `MERGE INTO` from Spark, with engine-dependent coverage elsewhere; plain Hive on Parquet does not, and emulating it is partition-level rewrite without isolation.

The reasoning

The spec calls it **optimistic concurrency**: an atomic swap of one table metadata file for another is the basis for serialisable isolation. A writer never locks the table while working. It writes data files and new metadata, then commits by asking the catalog to replace the current metadata location **only if it is still the one it started from**. The catalog is what makes that compare-and-swap atomic — a conditional update in the Hive metastore, Glue, JDBC or a REST catalog server.

**The loser retries with validation.** The reliability docs describe commits as assumptions and actions. The append's assumption is trivial — nothing it adds can conflict — so it re-applies its new manifest on top of snapshot 42 and commits. The compaction replaced `file_a` and `file_b` with `merged.parquet`; it is only safe to retry if both files are still in the table. If the append merely added files, they are, and the compaction retries. If a concurrent delete had removed `file_a`, the compaction must fail. Retries are bounded by `commit.retry.num-retries` (default 4) and a total timeout.

**Isolation level decides which conflicts matter.** Row-level operations validate more: a `MERGE` in serialisable mode fails if another commit added files that might match its condition since it started; snapshot isolation relaxes that to conflicting deletes. Two `MERGE`s touching the same partition will conflict, and the loser has to redo its work.

**Why retries are usually cheap.** Appends write a new manifest that can be reused across attempts, and sequence numbers are inherited so only the manifest list has to be rewritten on retry. The expensive case is a long rewrite losing to many small commits — a common reason to scope compaction by partition and enable `partial-progress.enabled`.

The reasoning

`hoodie.write.concurrency.mode` has three values in the 1.2 docs. `SINGLE_WRITER`, the default, maximises throughput and assumes nobody else writes. Table services still run alongside the writer under MVCC, so compaction and cleaning do not count as a second writer in that sense.

**OCC** allows several writers with conflict checks at commit time. It needs `hoodie.write.lock.provider` — storage-based, ZooKeeper, Hive metastore, DynamoDB, file-system, or in-process for single-writer use. Conflicts are detected at **file level**: if two writers touched the same file groups, the later one fails and has to retry. Two jobs upserting the same hot keys will keep colliding.

**NBCC**, introduced in Hudi 1.0, lets multiple writers commit without failing. Each writes its own log files into the same file groups; ordering by completion time plus the ordering field lets compaction and readers merge them deterministically. The docs restrict it to **merge-on-read** tables with a **simple bucket index or partition-level bucket index**, and it is not yet supported with clustering. Those restrictions are why it suits multi-stream ingestion into one MOR table.

Compared with Iceberg: Iceberg's optimistic commit needs no extra lock service because the catalog swap is atomic; Hudi's OCC depends on the lock provider being right. A misconfigured or in-process lock with two real writers is how Hudi tables end up with inconsistent file slices.

The reasoning

A snapshot is a list of files. `SELECT … TIMESTAMP AS OF '2026-06-01 00:00:00'` or `VERSION AS OF <snapshot id>` finds the snapshot current at that moment and plans against its manifests. Nothing is reconstructed or replayed — the old files simply still exist.

**Where the cost comes from.** A copy-on-write MERGE that updates 5,000 rows scattered over 300 files writes 300 new files and removes the old 300 from the current snapshot. The old 300 are still referenced by the previous snapshot, so they stay. Every hour, another set. After 90 days the table holds the live data plus every version of every file rewritten in that window. A table whose updates touch a small fraction of its files grows modestly; one whose MERGE rewrites much of it each hour can hold many times its live size. The number depends on churn, so measure it rather than guess: the `snapshots` and `files` metadata tables show what each snapshot added and removed.

**How it ends.** `expire_snapshots` removes snapshots older than `older_than` (default 5 days in the procedure, from `history.expire.max-snapshot-age-ms`), keeping at least `retain_last` (default 1), and deletes files no remaining snapshot references. Merge-on-read changes the shape of the cost — small delete files instead of rewritten data files — but compaction then rewrites data, which creates the same kind of garbage.

**Delta and Hudi are the same idea with different knobs.** Delta keeps removed files until `VACUUM` deletes those older than the retention window (7 days by default). Hudi keeps old file slices until the cleaner removes them; `hoodie.clean.commits.retained` (default 10) bounds how far back time travel and incremental reads can go. In all three, a long time-travel window is a storage decision, and should be priced as one.

The reasoning

**Delta.** Each commit is `_delta_log/000…500.json`, listing actions: add file, remove file, metadata change, protocol change. Periodically the log is summarised into a Parquet checkpoint so readers do not replay every commit from zero. Version 500 is "the checkpoint at or below 500, plus the JSON commits after it". The table is the log.

**Iceberg.** Each commit writes a new `metadata.json` holding the schema history, partition specs and the list of snapshots. Snapshot 500 points to its manifest list, which points to manifests. A reader of version 500 does not replay anything; it reads that snapshot's tree. The catalog stores only the pointer to the current metadata file.

**Schema evolution.** Iceberg assigns every column a unique field id and data files are read by id, so rename, drop and reorder are metadata-only and a new column with an old name never picks up old data. Delta by default resolves columns by name in Parquet; renaming or dropping a column requires **column mapping** (`delta.columnMapping.mode = 'name'`), which records a physical name per column. Once enabled, Delta behaves similarly — but it is a table feature you have to turn on, and it upgrades the protocol.

Where they meet: both keep old data files until an explicit clean-up (`VACUUM` or `expire_snapshots`), both support time travel by version or timestamp, and Delta UniForm can generate Iceberg metadata for a Delta table so Iceberg readers can read it. That interoperability is read-only from the Iceberg side.

The reasoning

In Hive, partitioning is a **column**: the table has `event_date`, writers must fill it, and readers must filter on it. The Iceberg docs list the failures — a writer uses the wrong format or time zone and results are silently wrong; a query filters on `event_time` but not `event_date` and scans the whole table because Hive does not know the two are related.

Iceberg partitions by **transforms of source columns**: `year`, `month`, `day`, `hour` for timestamps, `bucket(N, col)`, `truncate(W, col)` and identity. The spec is metadata, so the writer computes `day(event_time)` for every row itself. On read, the planner knows the relationship, so a predicate on `event_time` becomes a range on the partition value and is used to prune manifests and files. The query above reads one day's files and uses column stats to narrow further.

The same property enables **partition evolution**: because queries never name partition values, the spec can change — from `day` to `hour`, or adding `bucket(16, customer_id)` — without breaking queries. Old files keep the old spec, new files use the new one, and planning handles both.

The limits: pruning only works on predicates the planner can map through the transform (a filter on `date_format(event_time, …)` may not prune), and a bad transform choice still hurts. `hour()` on a low-volume table produces tiny files; `bucket()` on a column you never filter or join on buys nothing.

The reasoning

`ALTER TABLE db.events REPLACE PARTITION FIELD day(event_time) WITH hour(event_time)` — or `ADD PARTITION FIELD` / `DROP PARTITION FIELD` — creates a new **partition spec** with a new spec id. The docs are explicit: old data written with an earlier spec remains unchanged, metadata for each spec is kept separately, and partition evolution does not eagerly rewrite files.

Every data file in a manifest records the spec it was written with. When a query filters on `event_time`, the planner evaluates the filter against each spec in turn — daily ranges for old files, hourly for new ones. Because users filter on `event_time`, never on a partition column, they do not notice the change.

**Why it matters.** Data volumes change. A table that was right with daily partitions at 5 GB a day may be wrong at 2 TB a day. In Hive the fix is a new table, a full copy and every query rewritten. In Iceberg it is one DDL statement, and you can rewrite older partitions later, if ever, with `rewrite_data_files` scoped by a filter.

**The caveats.** Old files keep the old layout, so queries over old data prune only as well as the old spec allows. Engines that bypass Iceberg and read the directory tree will be confused by mixed layouts — another reason to read only through the catalog. And each spec change is permanent history in the metadata; changing it weekly is a smell.

The reasoning

The evolution docs list supported changes: **add**, **drop**, **rename**, **update** (widen a type, such as `int` to `long` or `float` to `double`), and **reorder**, including inside nested structs. All are **metadata changes** — the new schema is written into the metadata file and data files are untouched.

What makes that safe is the **field id**. Each column has an integer id unique in the table, stored in the Parquet files' schema. A reader maps the table schema to file columns by id, not by name or position. Dropping `discount` (id 7) removes id 7 from the current schema. Adding a new `discount` assigns id 12. Old files contain a column with id 7 and none with id 12, so the new column reads as null (or its default in v3) for them. The docs spell out the failure this prevents: formats that track by name can un-delete a column when a name is reused.

Renames work for the same reason: the id stays, only the name in the metadata changes, and old files still resolve. Format v3 adds **default values** — `initial-default` for existing rows and `write-default` for new ones — so a new non-null column can be added without rewriting data.

What is not allowed: narrowing a type, changing `string` to `int`, or anything else that could make existing values unreadable. Those need a new column and a backfill. Note also that older time-travel reads use the schema of the snapshot you travel to, unless the engine is asked otherwise.

The reasoning

**What it stores is small.** For `prod.sales.orders` the catalog essentially stores the current `metadata.json` location, plus namespace information. Everything else — schema, snapshots, partition specs — is in the metadata file on object storage. The catalog's job is to find that file and to make commits atomic by swapping the pointer only if nobody else has.

**The implementations.** A **Hive metastore** catalog stores the pointer as a table property and uses the metastore's locking. **AWS Glue** stores it in the Glue Data Catalog and uses Glue's conditional update for the swap. **JDBC** uses a database row. **Nessie** adds git-like branches and commits across many tables, so you can commit changes to several tables together or work on a branch. A **Hadoop** catalog uses file-system renames and is not safe on object stores that do not provide atomic rename.

**Why REST.** Every catalog above needs a client implementation in every engine. The REST catalog, specified as an OpenAPI document in the Iceberg repository, moves that logic to a server: engines speak one protocol, and the server can do commit validation, credential vending and access control centrally. Apache Polaris, Unity Catalog, Lakekeeper and AWS's Glue and S3 Tables endpoints are examples of implementations.

**The practical rule.** One table, one catalog. Two engines configured with different catalogs pointing at the same table location will each have their own idea of the current metadata file, and each will overwrite the other's commits. Most "Iceberg lost my data" incidents are two catalogs, not a bug.

The reasoning

Every write is an instant on the timeline, and each file slice records which instant produced it. An **incremental query** asks for records changed between two instants. In Spark SQL: `SELECT * FROM hudi_table_changes('trips', 'latest_state', '20260301101500000')` returns the latest state of each record changed after that time; in the DataFrame API it is `hoodie.datasource.query.type = incremental` with a begin instant.

**The pattern.** The downstream job keeps a checkpoint — the last completed instant it consumed — reads from there, merges the result into `fact_trips`, and advances the checkpoint only after its own commit succeeds. Hudi 1.0 and later use **completion time** for incremental ranges, which avoids a long-running commit that started earlier but finished later being skipped.

**Latest state vs CDC.** `latest_state` gives the current version of changed rows, which is enough to re-merge them. With `hoodie.table.cdc.enabled` the table logs change data and `hudi_table_changes('trips', 'cdc', …)` returns operation type plus before and after images — needed when downstream must subtract the old value, as in running totals.

**The limit is retention.** Incremental reads only work while the files for the range still exist. If the cleaner has removed them (it keeps `hoodie.clean.commits.retained`, default 10 commits) or instants have aged out, the job has to fall back to a snapshot read and a full reconciliation. A consumer that falls behind by more than the retention window cannot catch up incrementally.

The reasoning

**Compaction** exists because of merge-on-read. Each upsert appends log blocks to a file group; snapshot queries merge them with the base file at read time. Compaction reads base plus logs and writes a new base file, producing a new file slice, after which the logs are no longer needed. It is scheduled on the timeline and triggered by `hoodie.compact.inline.trigger.strategy` (`NUM_COMMITS`, `TIME_ELAPSED`, `NUM_AND_TIME`, `NUM_OR_TIME`, …) — inline with `hoodie.compact.inline = true`, or asynchronously in a separate job or thread, the default pattern for streaming writers.

**Clustering** exists because ingestion produces files that are too small and in arrival order. It groups small files per the plan strategy (`hoodie.clustering.plan.strategy.small.file.limit`, `...target.file.max.bytes`), optionally sorts by `hoodie.clustering.plan.strategy.sort.columns` (linear, z-order or Hilbert layout), and writes new file groups, committing a `REPLACE_COMMIT` that swaps them for the old ones. Inline (`hoodie.clustering.inline`) or async (`hoodie.clustering.async.enabled`).

So at 288 commits a day: compaction keeps snapshot-read latency bounded by limiting how many log blocks accumulate per file group; clustering keeps file counts and data locality sane, which compaction does not change — it rewrites a file group's base file, not the number or order of files. There is also **log compaction**, which merges small log files together without rewriting the base.

One interaction worth knowing: with non-blocking concurrency control, clustering is not yet supported per the 1.2 docs, and clustering on a table with frequent upserts must avoid conflicting with the writer on the same file groups.

The reasoning

Iceberg does not store data in a format of its own; data files are usually Parquet, ORC or Avro. What Iceberg defines is the **table**: a metadata file with the schema, partition specs and snapshots, each snapshot pointing through a manifest list to manifests that list data and delete files with statistics. A catalog stores the pointer to the current metadata file and swaps it atomically on commit.

From that come the features: **serialisable commits** by optimistic pointer swap, **time travel and rollback** by reading or restoring an older snapshot, **hidden partitioning** and **partition evolution** because partitions are transforms in metadata, **schema evolution** by field id, and fast planning because files are found through metadata rather than listing.

It was built at Netflix and is an Apache project, with formats v1 (analytic tables), v2 (row-level deletes) and v3 (deletion vectors, row lineage, variant and geospatial types, default values) adopted in the spec, and v4 in development. It is engine-neutral by design: Spark, Flink, Trino, Presto, Hive, Snowflake, BigQuery and others read and many write it.

The reasoning

Hudi came from a specific problem: applying a continuous stream of database changes to very large tables on a lake without rewriting partitions every run, and letting the next job consume just those changes. The name — Hadoop Upserts Deletes and Incrementals — says so.

Its building blocks: a **timeline** of instants under `.hoodie/timeline` that makes writes atomic and records table services; **record keys** and an **index** that map each key to a file group so an upsert touches only affected files; **copy-on-write** and **merge-on-read** table types; **incremental** and **CDC** queries driven by the timeline; and **table services** — compaction, clustering, cleaning, indexing — that run inline or asynchronously.

Current versions are the 1.x line (1.2 at the time of writing). 1.0 introduced an LSM-based timeline history, non-blocking concurrency control for merge-on-read tables, and secondary and expression indexes; 1.1 added partitioned record-level and partition-level bucket indexes and a pluggable table-format framework. It is most often chosen for high-frequency upsert and CDC workloads.

The reasoning

**Copy-on-write:** to change one row, read the file containing it, write a new file with the change, and commit the swap. Cost is proportional to file size, not change size. Readers read clean columnar files with no merge work.

**Merge-on-read:** write only the change. In Iceberg (v2 and above) that is a position delete, an equality delete or, in v3, a deletion vector, plus new rows in new files; set per operation with `write.delete.mode`, `write.update.mode` and `write.merge.mode`, all defaulting to `copy-on-write`. In Hudi it is log blocks appended to the file group of a `MERGE_ON_READ` table. Readers combine base data with pending changes, and cost grows until **compaction** folds them in.

**How to choose:** the ratio of reads to updates and how updates are spread. Few, large, batched updates on a heavily read table favour copy-on-write. Frequent small updates spread across many files — CDC, GDPR deletes, late corrections — favour merge-on-read, provided compaction is scheduled. Delta Lake's deletion vectors are the same merge-on-read idea for deletes.

The reasoning

Syntax: Iceberg in Spark uses `SELECT * FROM t TIMESTAMP AS OF '2026-03-01 00:00:00'` or `VERSION AS OF <snapshot id | branch | tag>`; Delta uses `VERSION AS OF` and `TIMESTAMP AS OF`; Hudi uses `TIMESTAMP AS OF` with an instant time.

Uses: compare the table before and after a suspect job; rebuild a report exactly as it was on its publication date; roll back — Iceberg's `rollback_to_snapshot` or `rollback_to_timestamp`, Delta's `RESTORE`, Hudi's savepoint and restore. Iceberg adds **tags** to keep a named snapshot beyond normal expiry, and **branches** to write and validate changes before publishing.

The limit is retention. Iceberg's `expire_snapshots` (default max age 5 days), Delta's `VACUUM` (7-day default retention) and Hudi's cleaner (`hoodie.clean.commits.retained`, default 10) remove the files old versions need. Time travel is a short-term safety net, not an archive; keep long-term history with tags or explicit copies.

The reasoning

A **data lake** is object storage holding files in any format, with nothing guaranteeing what a "table" contains at any moment. A **warehouse** stores data in its own format behind its own engine, with transactions and governance built in. A **lakehouse** keeps the lake's storage and adds a table format in between.

The table format is what changes the guarantees: an atomic list of files per version, schema enforcement and evolution, row-level `MERGE` and `DELETE`, time travel, and statistics for pruning. A catalog makes tables discoverable and is where access control is increasingly applied.

The trade-offs are operational: someone has to run compaction, snapshot expiry and orphan-file clean-up that a warehouse does internally, and multi-engine access only works if every engine goes through the same catalog.