D8LooP - Practise Data Engineering

01 · Two loose threads

Two Loose Threads, One YAML File

when did you last update?

⏱ 4 min · Topic 1 of 10

Chapter 4 left fct_subscription_revenue reaching straight into raw.stripe.payments — a mart touching a source with no staging model between them. That is one loose thread. The other: every model since Chapter 3 has used ref() or source() instead of a hard-coded name, and Chapter 3 priced six things that habit buys you.

Declaring a source instead of hard-coding a raw table buys the same six things ref() buys a model — plus a seventh that exists nowhere else. Predict the eight statements below: seven capabilities plus one control, then check yourself.

Two ways to read stripe.payments — predict what a hard-coded reference loses

models/staging/stg_stripe__payments.sql — hard-coded

from raw.stripe.charge

models/staging/stg_stripe__payments.sql — declared source

from {{ source('stripe', 'payments') }}

01The SELECT compiles and returns the right rows, today

02dbt computes a safe build order for it automatically (Chapter 3: build order)

03dbt can safely parallelize it against other models (Chapter 3: parallelism)

04It shows up in lineage graphs and the generated docs site (Chapter 3: lineage and docs)

05It is reachable by node selection — e.g. --select source:wavelength_app+ (Chapter 3: --select model+)

06The address resolves correctly in dev, CI, and prod without editing the SQL (Chapter 3: environment portability)

07A breaking upstream change gets caught by CI before merge (Chapter 3: CI impact analysis)

08dbt can tell you the data is late, on a schedule, before any test runs

Predict every row, then commit — the score and the reasoning appear together.

Core mental model

sources.yml is a passport for a table dbt does not own. It does not change the table itself; it lets dbt recognize it, place it in the graph, and ask it a question a hard-coded name can never be asked: when did you last update?

Why it matters

A declared source is the only place dbt's dependency graph, docs, node selection, and freshness checks can see a raw table at all. Skip the declaration and every one of those capabilities doesn't break — it silently never existed for that table in the first place.

Source: A raw table dbt reads but never builds — declared once in sources.yml so the rest of the project can refer to it by name instead of by address.
source(): The Jinja function that resolves to a declared source's fully-qualified address at compile time, and records a dependency edge exactly the way ref() does for a model.
Dependency edge: The graph link dbt draws only when a model calls ref() or source() — recorded once at parse time, and the one thing a hard-coded table name never creates.

Common mistake

Conflating "this view is fresh" (Chapter 4's materializations) with "this source is fresh" (this chapter). A view can be freshness-perfect — zero lag from its own inputs — while the source underneath it is hours stale. A source can pass freshness while sitting under a table that has not rebuilt in a day. They are different axes entirely, and the same word describes both.

Better habit

Declare every raw table you read as a source before writing the first model against it — not after something breaks.

When a raw table changes name or location, expect to edit exactly one YAML entry, never a SQL file.

Read a project's sources.yml before its models — it names every raw table the project depends on, in one file, without grepping SQL for hard-coded schemas.

The raw layer still never changes

Chapter 1 already made the case that raw data is immutable and untouched by dbt — this chapter is not about touching it, only about telling dbt it exists. The load itself (the "EL" that puts these rows in raw.stripe and raw.wavelength_app before dbt ever runs) is upstream of everything here.

ETL vs ELT (Data Pipeline KB)

Remember this

A source declaration is not decoration on a raw table name — it is the only thing that makes that table visible to dbt's graph, docs, selection, and freshness checks at all.

Sources, Freshness, and the Staging Layer