Analytics Engineering · Knowledge Base

dbt

The transformation layer most warehouses are actually built on. From your first model to a governed multi-project mesh: how ref() builds a DAG, what each materialization really costs, incremental models that stay correct, tests and contracts that catch breakage before a dashboard does — and the production failures that ship wrong numbers while every run stays green.

22 chapters4 levelszero → advanced
Start
01

Foundations

5 chapters
02

Building a Project

6 chapters
06

Jinja, Macros, and the Compilation Model

What happens between your SQL file and the query the warehouse runs: the Jinja context, the parse/compile/run phases, macros, and how to debug them.

Jinja contextMacrosexecute flagrun_queryadapter.dispatch
Read chapter →
07

Tests: Generic, Singular, and Custom

The SQL every built-in test compiles to, severity and thresholds, writing your own generic test — and the wrong numbers that no test will ever catch.

Built-in testsSeveritySingular testsCustom genericsstore_failures
Read chapter →
08

Unit Tests: Proving the Logic Before the Data Arrives

Fixture-driven tests that run without warehouse data, so dedup rules, null handling, and window boundaries are proven before the model ever ships.

unit_tests YAMLgiven/expectFixture formatsMocking refsUnit vs data test
Read chapter →
09

Incremental Models Done Right

is_incremental(), unique keys, and all five strategies compared on real row counts — plus late-arriving data, schema change, and the drift nobody notices.

is_incremental()merge vs insert_overwritemicrobatchon_schema_changeDrift
Read chapter →
10

Snapshots and Slowly Changing Dimensions in dbt

Capturing history from a source that overwrites itself — strategies, the generated validity columns, hard deletes, and why a snapshot can never be rebuilt.

timestamp vs checkdbt_valid_from/toHard deletesSCD2Irreversibility
Read chapter →
11

Seeds, Documentation, and Exposures

Static lookups that belong in git, docs that stay accurate because they live beside the model, and lineage that reaches all the way to the dashboard.

Seedsdocs generateDoc blocksExposuresImpact analysis
Read chapter →
03

Structure & Production

6 chapters
12

Project Structure: Staging → Intermediate → Marts

The layering convention and the reasoning behind it, one grain per model, and how to decompose an 800-line query without changing a single row.

LayersGrainFan-out trapsNamingRefactoring
Read chapter →
13

Contracts, Constraints, Versions, and Access

Making a model a promise: enforced column types, constraints your warehouse may or may not honour, versioned migrations, and who is allowed to ref you.

Enforced contractsConstraintsModel versionsaccess & groupsDeprecation
Read chapter →
14

Environments, CI/CD, and Slim Runs

Dev, CI, and prod without editing code; a throwaway schema on every PR; and slim CI that builds only what changed and defers the rest to production.

Targetsgenerate_schema_namestate:modified--deferdbt clone
Read chapter →
15

Orchestrating dbt: Airflow, dbt Cloud, and Cosmos

One task or forty: how dbt gets invoked in production, what each option costs in parse time and blast radius, and what a half-finished run means downstream.

dbt Cloud jobsAirflowCosmosdbt retryPartial failure
Read chapter →
16

Performance and Cost

Why sixteen threads is not sixteen times faster, finding the model that owns your critical path, and the warehouse knobs that actually move the bill.

Threads & critical pathrun_results timingWarehouse sizingPartition pruningCost attribution
Read chapter →
17

Debugging and Production Failure Modes

A symptom-to-fix runbook for analytics engineering — including the worst failure of all: the run that goes green while the numbers are wrong.

Reading logsArtifacts as datadbt retrySilent failuresOn-call triage
Read chapter →
04

Scale & Expertise

5 chapters