Iceberg, Hudi & Table Catalogs

A table format turns a folder of Parquet files into a real table. You met the idea in "Storage Layers & File Formats". You met Delta Lake in depth in the Apache Spark section.

This chapter looks inside the other two formats, Apache Iceberg and Apache Hudi. Then it covers the catalog, the part that lets many engines share one table.

What you already know

• A table format keeps a log or metadata that lists the files in the table.
• A commit is all or nothing. A reader sees the old version or the new one.
• Time travel reads an old version, as long as its files still exist.
• Copy-on-write rewrites a file to change a row. Merge-on-read writes a small change file and merges it when reading.

If a line in this list is new, read "Storage Layers & File Formats" first. For Delta, read "Delta Lake & the Lakehouse".

What you will learn

1. The files inside an Iceberg table, and how a query uses them.
2. How time travel works without copying data.
3. What happens when two jobs write at once.
4. Hidden partitioning, and changing partitions without a rewrite.
5. The jobs that keep a table healthy.
6. Inside a Hudi table: the timeline, keys and table types.
7. Catalogs, and how Spark, Trino and Snowflake can share one table.
8. How to choose a format, and when a warehouse is better.

We use the shop table orders from "Storage Layers & File Formats", now as an Iceberg table. The snapshot numbers are made up.

An Iceberg table is a tree of small files that point at the data. A reader starts at the top and walks down.

This table lists the levels, from the top:

A snapshot is the full list of data files at one commit. Here are the three commits to orders:

A commit never edits an old file. It writes new data files, a new manifest, a new manifest list and a new metadata file. Then the catalog points at the new metadata file.

How a query uses the tree

Here is a query for one day:

SELECT SUM(amount) AS revenue
FROM   shop.orders
WHERE  placed_at >= TIMESTAMP '2026-01-22 00:00:00'
  AND  placed_at <  TIMESTAMP '2026-01-23 00:00:00'
  AND  status = 'paid';

Listing a folder on object storage is slow when it holds millions of files. Iceberg was built at Netflix partly to avoid that listing.

Snapshot 1002 still lists files a and b. File b was not deleted when 1003 removed it from the table. So you can read the table as it was:

-- By snapshot id
SELECT SUM(amount) AS revenue
FROM   shop.orders VERSION AS OF 1002
WHERE  placed_at >= TIMESTAMP '2026-01-22 00:00:00'
  AND  placed_at <  TIMESTAMP '2026-01-23 00:00:00'
  AND  status = 'paid';

-- The list of snapshots is a table too
SELECT snapshot_id, parent_id, operation
FROM   shop.orders.snapshots;

You can also read by time with TIMESTAMP AS OF. Iceberg uses the snapshot that was current at that moment.

Delta does the same with a version number in its _delta_log folder. Hudi does it with the time of a commit on its timeline.

Object storage has no locks for a table. So Iceberg uses optimistic concurrency: each writer works alone, and checks for a clash only at the end.

The check happens in the catalog. A writer asks: "Swap the pointer from metadata 3 to my metadata 5, but only if it still points at 3."

Here two jobs start from snapshot 1003 at the same time. Job A appends orders for 2026-01-23. Job B runs a MERGE on 2026-01-22:

Both changes are in the table, and nobody waited for a lock. Step 4 is the important one.

If A had changed the same rows that B read, the check fails. B's commit fails, and the job must run again.

Delta works the same way, with its log in place of the catalog swap. The Spark chapter "Delta Lake & the Lakehouse" shows which Delta operations clash.

In an older Hive-style table, the partition is a separate column, such as dt = '2026-01-22'. Every writer must fill it, and every query must filter on it.

If an analyst filters on placed_at and forgets dt, the engine reads every partition. Nothing fails, the query is just slow and costly.

Iceberg stores a rule instead of a column. The rule is a transform of a real column, such as day(placed_at):

CREATE TABLE shop.orders (
  order_id    BIGINT,
  customer_id INT,
  status      STRING,
  amount      DECIMAL(10, 2),
  placed_at   TIMESTAMP
) USING iceberg
PARTITIONED BY (day(placed_at));

Other transforms exist: hour(), month(), year(), bucket(16, customer_id) and truncate(4, sku). A bucket transform spreads keys over a fixed number of groups.

The shop grows. One day of orders is now too big, and most queries ask for the last few hours. You want hourly partitions.

In a Hive-style table you would rewrite all the old data. In Iceberg it is one metadata change:

-- Needs the Iceberg SQL extensions in Spark
ALTER TABLE shop.orders
REPLACE PARTITION FIELD placed_at_day WITH hour(placed_at);

A query across both periods still works. Iceberg plans the old files with the old rule and the new files with the new rule.

If you also want old data in hourly files, run rewrite_data_files on it later, as a normal background job.

A table format never deletes or merges files by itself. If nobody runs the upkeep jobs, the table gets slower and bigger every day.

Streaming writes make this fast. Say a job commits every minute, with 10 writers. That is 10 files a minute, or 14,400 small files a day.

Iceberg gives four Spark procedures for upkeep:

Run them in this order, for example every night:

CALL lake.system.rewrite_data_files(table => 'shop.orders');

CALL lake.system.expire_snapshots(
  table       => 'shop.orders',
  older_than  => TIMESTAMP '2026-01-15 00:00:00',
  retain_last => 10);

CALL lake.system.remove_orphan_files(table => 'shop.orders', dry_run => true);

CALL lake.system.rewrite_manifests('shop.orders');

Run remove_orphan_files with dry_run first and read the list. A file that a running job has just written can look like an orphan.

Delta has the same jobs with other names: OPTIMIZE for compaction, and VACUUM for deleting old files.

Apache Hudi was built at Uber for frequent updates, such as a database copied in with CDC. Its design starts from the key of each row.

The timeline

Hudi keeps a timeline in the .hoodie folder. It records every action on the table, with a time and a state.

Each action moves through three states: REQUESTED, INFLIGHT and COMPLETED. Readers use only completed actions. This is a short timeline for orders:

A failed write stays INFLIGHT. Hudi later runs a ROLLBACK action that removes its partial files.

Keys and the ordering field

Every Hudi row has a record key, such as order_id. An index maps each key to the file group that holds it, so an update touches only that file group.

Two versions of one key can arrive in the same batch. The ordering field decides which one wins. It was called the precombine field in older versions.

hoodie.datasource.write.operation    = upsert
hoodie.datasource.write.recordkey.field = order_id
# ordering field: updated_at (the larger value wins)

Table types and query types

A read-optimized query reads only the base files. It is faster, but it misses changes not yet compacted.

An incremental query returns only the rows that changed after a given time. A downstream job can use it to read only what is new.

A catalog maps a table name to its current metadata. For Iceberg it also does the pointer swap of every commit, so the catalog decides which commit wins.

These are the catalogs you meet most:

The Iceberg REST catalog is a standard web API for catalogs. Any engine that speaks it can use any catalog that serves it.

So one table can have one writer and many readers, in different engines:

Every engine must commit through the same catalog. If two catalogs both think they own the table, each one moves its own pointer, and the table splits into two histories.

Formats can also share tables. Delta UniForm writes Iceberg metadata next to the Delta log, so Iceberg readers can read a Delta table. The Apache XTable project translates metadata between the three formats.

All three formats give atomic commits, MERGE and time travel. Choose from your engines, your writes and your team:

Sometimes the right answer is no lake at all. A plain warehouse table is often better when:

• The data is small enough that one warehouse handles it well.
• Only one engine, the warehouse, reads and writes it.
• The team works in SQL and has nobody to run upkeep jobs.

A lakehouse saves money and avoids lock-in at large scale, but it adds work: catalogs, upkeep and file sizes. Say both sides in an interview.

These are the table format questions senior loops ask most, with the short answer and the card that goes deeper:

Build each scenario on the pipeline canvas with the chapter closed. In each one, say which table format you use, how it is partitioned, and which upkeep jobs run.