What You'll Master Here
records together for fetching whole records, columns together for scanning a few fields over many rows. The match is the whole game.
Where and how data physically sits on disk is not an implementation detail. It is one of the biggest levers on cost and speed in the whole pipeline: the same query, over the same data, returning the same answer, can cost $1.05 or $0.0016 purely because of the file format.
A supermarket is the everyday version. A shopping basket keeps one customer’s items together — perfect for checking out that customer, useless for answering "how many bananas did we sell today?". The stockroom keeps all the bananas together instead. Same goods, opposite arrangement, and each is hopeless at the other’s job.
The single idea underneath everything here is that storage should match the access pattern. Operational systems fetch whole records, so they keep records together. Analytics scans a few columns over millions of rows, so it keeps columns together.
By the end you will know why columnar formats dominate analytics and be able to derive the saving yourself; what is actually inside a Parquet file; what a data lake physically looks like and what a catalog adds; how a table format makes cheap files behave like a database table; and how to choose between copy-on-write and merge-on-read without guessing.
Storage should match the access pattern: records together for fetching whole records, columns together for scanning a few fields over many rows. The match is the whole game.
Storage format determines how much data a query must read, which drives both speed and cost. Choosing the right layout is routinely a bigger win than any amount of query tuning, and it is a one-off decision rather than a permanent effort.
- OLTP
- Online Transaction Processing — the operational systems that run the business: taking an order, updating a balance. They read and write whole records, constantly, one at a time.
- OLAP
- Online Analytical Processing — the analytical systems that answer questions about the business. They scan a few columns across millions of rows and rarely touch a single record.
- row storage
- Keeping all of a record’s fields together on disk. Right for OLTP.
- columnar storage
- Keeping each column’s values together on disk. Right for OLAP.
- object storage
- Cheap, effectively infinite cloud file storage — S3, GCS, Azure Blob. Files can be created and deleted but never edited in place, which turns out to shape everything in this chapter.
- data lake
- Large-scale storage of files on object storage, queried in place by whatever engine you like.
- lakehouse
- A lake plus a table format, so the cheap files behave like real database tables — transactions, schema management, time travel.
Storing analytical data in a row format and wondering why scans are slow. Every query reads every column of every row, scanning far more data than it needs — slowly, and at full price.
Assuming every warehouse bills per byte scanned. Athena and BigQuery do, so waste is visible on the invoice. Snowflake and Redshift bill for compute time, where the same wasted bytes show up as longer queries and a bigger cluster — the cost is identical, it is just harder to see.
Match storage layout to the dominant access pattern.
Use columnar formats (Parquet) for anything analytical.
Add a table format when a lake needs updates, transactions or schema management.
Analytics reads a few columns across many rows. Columnar storage makes that read only those columns — and then, because like sits next to like, compresses them far harder than a row format ever could. Two wins from one decision.
The next topic puts a price on the whole subject before explaining any of it. Work out the annual difference yourself first; everything after it is the explanation of a number you have already seen.
Storage layout and format are the largest levers on analytical cost and speed; match layout to access pattern, and add a table format when cheap files need to behave like tables.
- Explain why columnar storage suits analytics and row storage suits OLTP.
- Define a data lake and a lakehouse in one sentence each.