A Project You Can Hold In Your Head
A layer is not a folder. It is a promise about grain and about who is allowed to change it. Staging promises one row per source entity from exactly one source. Intermediate promises to declare its grain, which may differ from its inputs’. Marts promise one row per business entity or event, with a human on the other end. The folder is just where you keep the promise.
Eleven chapters in, Wavelength has 22 graph nodes, five marts in one flat folder, and an intermediate layer with exactly one model in it. Nothing is broken. Every run is green — except one test, and it is green for the wrong reason.
Click around the x-ray below before reading anything else. Switch the lens, switch the view, and find out for yourself what a new hire would find on their first morning.
A layer is not a folder. It is a promise about grain and about who is allowed to change it. Staging promises one row per source entity from exactly one source. Intermediate promises to declare its grain, which may differ from its inputs’. Marts promise one row per business entity or event, with a human on the other end. The folder is just where you keep the promise.
Folder layout looks like a taste question, and it is not. It is the question of where the row count is allowed to change. In a 437-line script the grain changes six times inside one file and none of those six moments has a name — so none of them can have a test, and none of them can be selected in CI. Split the same logic across staging, intermediate and marts and every one of those moments becomes a named, queryable, testable relation. That is the whole engineering argument for layering, and it has nothing to do with tidiness. Everything else in this chapter is a consequence of it.
- grain
- What one row of a table means — one payment, one subscription version, one show — expressed as the exact set of columns that is unique per row. Every layer boundary in this chapter is a place where the grain is allowed to change, and every one of those places should be named.
- staging
- One model per source table, 1:1 with it, reading exactly one source() and joining nothing. Chapter 5 built this layer and owns the argument for it; this chapter only defends the boundary against models that want to sneak in.
- intermediate
- Models that reshape for reuse: joins, pivots, aggregations to a join grain. Named after the verb they perform, not exposed to consumers. This project has exactly one, which is Section 1’s second finding.
- mart
- The consumable layer. dbt Labs’ marts guide defines it by grain rather than by prefix: a mart “represents a specific entity or concept at its unique grain”. fct_ and dim_ are a style layered on top of that sentence, and Section 7 shows two dbt Labs artifacts disagreeing about them.
- root model
- A model that calls neither ref() nor source(). It builds fine and is invisible to lineage, to state:modified+, and to Chapter 11’s +exposure:* traversal. dbt-project-evaluator lints it as fct_root_models. Section 6 takes one apart.
- user_id vs listener_id
- Wavelength’s app calls this account user_id (stg_listens, values like u_4); its billing system calls the SAME account listener_id (subscriptions, snap_subscriptions, values like L-9001). One entity, two systems, two names — and the raw values do not line up, so a join needs a resolved key, not a rename. stg_subs is the one relation that has ever carried both on the same row, because wavelength_app.subscriptions carries user_id as a foreign key back to the app account alongside its own listener_id — declared in sources.yml this chapter. That is where the crosswalk lives, and Section 3 shows it.
| Layer | The promise about grain | Who may change it | What you can assert |
|---|---|---|---|
| staging | One row per source entity. One source, no joins. | Anyone — freely. Nothing downstream depends on its history. | row count equals the source’s row count; unique on the natural key |
| intermediate | I declare my grain, and it may differ from my inputs’. | The owner of the reuse. Two consumers means two people to ask. | a uniqueness test on whatever grain the model chose |
| marts | One row per business entity or event, and a human consumes me. | Nobody, casually. There are exposures pointing at this. | grain, plus every business rule you can name |
| snapshot | One row per entity per VERSION. Its history is irrecoverable. | Effectively nobody. You cannot re-derive what you did not capture. | at most one open row per key; non-overlapping intervals |
The fourth row is why Section 4 refuses to re-point stg_subs at snap_subscriptions. A snapshot is not a staging model with extra columns; it is a different grain with a different change-control regime, and giving it a stg_ name tells every future reader the opposite of both.
Treating project structure as a style question to settle after the models work. By then the structure IS the models. Moving a model between folders changes its config inheritance, its schema on some projects, and every path in every properties file that mentions it — and it is a rename to everything outside the graph that reads the warehouse directly. The cheapest moment to place a model is before it has consumers, and that moment lasts about a week.
Reading “green run” as “correct project” — or a passing transcript as a test’s history. This project has been carrying a uniqueness test on the wrong grain since Chapter 7 — it has been failing since the day it was written, because S-101’s second payment (pay_006) was already in the data Chapter 4 published. Chapter 11’s docs-tour transcript renders it as PASS; that is a recorded erratum in Chapter 11, not a fact about this test’s history. Section 2 shows the failure and names the fix. A transcript tells you what a run captured, not what an assertion has actually been asserting.
When you inherit a project, open the folder tree and the DAG side by side. Anything in the tree with no counterpart in the DAG, or vice versa, is where the interesting problems are.
Ask of every model: what does one row mean, and where is that written down? If the answer is “in someone’s head”, you have found the next test to write.
Place a new model in a layer on the day you create it, and say out loud which promise you are making. It is a thirty-second conversation now and a migration later.
Grain is what one row of a table means — one payment, one subscription version, one show — and the grain is exactly the set of columns that is unique per row. Join a table to a finer-grained one and the coarse rows repeat, so any sum taken afterwards is inflated by the fan factor; the cures are to aggregate before joining, or to sum only measures that already live at the joined grain. Declaring grain, additivity, mixed grain, and the fan and chasm traps by name are the Data Modeling KB’s, and it is worth reading before you draw a layer boundary. This chapter uses those ideas to decide which MODEL a multiplying join lives in and which LAYER the fix belongs to — never to re-derive the theory.
Grain: what one row means — Data Modeling KBRaw and bronze are roughly the sources Chapter 5 declared, staging plus intermediate is roughly silver, and marts are roughly gold — but the mapping is loose and the names are not interchangeable, because medallion is about data-quality tiers in storage while staging → intermediate → marts is about how code is organised inside one warehouse. The Data Pipeline KB owns bronze/silver/gold properly; this chapter does not teach it.
The data pipeline lifecycle — Data Pipeline KBThis chapter runs at 2026-07-27 09:00 UTC, after Chapter 10’s snapshot runs and Chapter 11’s docs work. It adds exactly one node — dim_subscription — taking the graph from 22 to 23, non-exposure nodes from 19 to 20, buildable nodes from 14 to 15, and manifest nodes from 23 to 24. It adds ZERO warehouse rows. The 1,000-row dataset in Sections 3 and 6 is a hypothetical, scaled-up stand-in for Wavelength’s real tables — Chapter 10’s 10-row snapshot and Chapter 9’s ~15 real listens — invented because fan-out is invisible at that scale, for the seven days ending 2026-07-18, and it is captioned as hypothetical every time it appears. Chapter 3’s 1.21 / 0.25 hours, Chapter 4’s 5 rows and $184.00, and Chapter 7’s 9-row stg_episodes all stand.
A layer is a promise about grain and about who may change it — and the convention’s real product is that every place the row count is allowed to change becomes a named, queryable, testable relation.