D8LooP - Practise Data Engineering

01 · Orientation

What You'll Master Here

Modern patterns govern how a warehouse is built and run; dimensional modeling still governs what the gold layer should look like. Cost is a modelling constraint, not a tuning afterthought.

⏱ 4 min · Topic 1 of 9

The classical methodologies (Kimball, Inmon, Data Vault) still decide how to model. What the cloud era changed is what a modelling decision costs: storage became cheap, compute became metered, and the shape you give a table is billed again on every query that touches it.

Work the ladder below before reading on. It is one dashboard question at Kestrel Mobile, a mobile network, and the four decisions this chapter teaches: which layer to read from, how much of history to rebuild, how the table is laid out, and how many columns the query pulls. Each is owned by a later topic, and each multiplies with the others rather than adding to them.

Core mental model

Modern patterns govern how a warehouse is built and run; dimensional modeling still governs what the gold layer should look like. Cost is a modelling constraint, not a tuning afterthought.

Why it matters

This is the stack data engineers work in today. Knowing how the modern patterns relate to dimensional modeling lets you build warehouses that stay cheap and trustworthy, instead of cargo-culting tools.

ELT: Extract, Load, then Transform inside the warehouse (vs ETL transforming first).
medallion: Layered refinement: bronze (raw) → silver (clean) → gold (curated).
dbt: A tool for building warehouse models as version-controlled, tested, dependency-aware SQL.
semantic layer: A single governed place where metrics are defined once for all tools.

One dashboard, four modelling decisionsKestrel Mobile shows “yesterday’s data usage by plan”. Every decision below changes what it costs to answer.

The four decisions

Which layer the dashboard reads
bronze.cdr_raw (JSON) — 18.0 TiB on disk
How much of history it rebuilds
all 24 months, every run — no reduction — reads every day
Whether the table is laid out by usage_dateone undivided table — a date filter still reads every byte
How many of the 40 columns it pullsall 40 (BI extract) — the extract drags every column along

Scanned by one refresh18.0 TiB

Per refresh · $112.50
Every 15 min, one day · $10,800.00
Priced at BigQuery’s published on-demand rate, $6.25 per TiB scanned. Snowflake and Databricks bill compute-time instead, so the same bytes arrive as a longer warehouse run rather than a line item — the modelling decision is identical either way.

select json_value(payload, '$.plan_code') as plan_code,
       sum(cast(json_value(payload, '$.bytes_used') as int64))
from bronze.cdr_raw
group by 1;

Reading the landing zone — $112.50 a refreshbronze.cdr_raw is one JSON document per record: no column can be skipped and no type is settled, so the engine parses everything it touches. Naming three columns changes nothing here. The layer you read from sets the floor for every other decision below it.

Common mistake

Thinking modern tools replace dimensional modeling. You build unstructured "just dump it in" warehouses; the gold layer still needs sound models.

Better habit

Load raw first, transform in the warehouse.

Refine data in explicit layers with tested, versioned SQL.

Define each metric once in a semantic layer.

The big idea

Modern patterns are about how you build and operate the warehouse; the modeling (stars, grain, SCDs) you learned still governs what the gold layer should look like.

Remember this

Modern warehouse patterns (ELT, medallion, dbt, OBT, semantic layer) are the operational context for dimensional modeling, not a replacement for it — and each one carries a cost you can read off the query.

Modern Warehouse Modeling