D8LooP
Start building

Build Your Foundation

Data Analyst to Data Engineer: How to Make the Move

Six moves, in the order they compound. Starts at the transformation layer, ends on call.

6 moves52 steps~16h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can model it, load it incrementally, schedule it, and own it downstream.

Made the move You build the tables you used to query

02

The plan, 6 moves

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Move 01 · ~2h

    Own the transformation layer

    You can turn a folder of ad-hoc SQL into a tested, version-controlled DAG.

    Skip this if you already ship dbt models with tests and a staging layer.

    1. SectionThe folder that stops scaling5 minYour SQL is fine. The folder it lives in is the problem.
    2. SectionWhere dbt sits: the T in ELT5 minWhere the analyst’s job ends and this one starts.
    3. SectionWhat a layer actually promises6 minStaging, intermediate, marts. Each owes something different.
    4. Exercisedbt: turn a raw payload into a staging model20 minin the studioRename, cast, clean, fix the grain. That is the whole staging job.
    5. SectionThe loop you will run every day6 minBranch, commit, push, review. Seven commands, then you are dangerous.
    6. Sectionref(): the one call that builds the graph6 minOne function turns a folder into a dependency graph.
    7. SectionFour built-ins, four anti-queries6 minA test is a query that should return nothing.
    8. ExerciseDuplicate snapshot keys12 minin the studioWrite the anti-query by hand. That is all a unique test is.
    9. ExerciseReferential integrity audit15 minin the studioAnd this is a relationships test. Orphans have nowhere to join.
    10. SectionGreen doesn’t mean right6 minGreen means nothing you tested was wrong. Not that it is right.
    11. Questionsdbt · Conceptual15 minTwenty questions on the layer you now own.

  2. Move 02 · ~3h

    Design the model instead of consuming it

    You can pick a grain, build the star around it, and defend both.

    Skip this if you already choose the grain before you draw any table.

    1. SectionThe same business, two different workloads6 minThe source system looks nothing like your warehouse. Here is why.
    2. SectionKimball’s four-step design process7 minFour questions in order, and the grain is question two.
    3. Worked exampleSubscription warehouse grain35 minin the studioA star with the grain declared. Read one before you draw one.
    4. ExerciseWatch time by title40 minin the studioReporting you have written a hundred times, now as a model.
    5. SectionConformed dimensions & the bus matrix6 minWhy two teams’ "customer" has to be the same customer.
    6. ExerciseCategory performance over time40 minin the studioReport across a dimension that changed. Your first versioning call.
    7. Exercisedbt: which row was current on the 15th?18 minin the studioThe newest row is the wrong answer. Read the interval instead.
    8. ExerciseInterview fanout-safe order revenue20 minin the studioProve the model does not double-count before anyone quotes it.
    9. QuestionsData Modeling · Model design15 minDesign questions, out loud. Say the grain first.

  3. Move 03 · ~3h

    Write Python you would let someone review

    You can write a transform as tested Python, not as a notebook cell.

    Skip this if you already ship Python with fixtures and a failure path.

    1. SectionEvery job is input → transform → output6 minThe shape of every job you will write. Learn it once.
    2. ExerciseTotal revenue by region25 minin the studioA GROUP BY you could write in your sleep, now in Python.
    3. SectionPlan the failure path first6 minThe half of the job SQL never made you think about.
    4. ExerciseValidate rows against a schema25 minin the studioBad rows arrive. Decide where they go before they do.
    5. ExerciseReconcile two systems row by row40 minin the studioTwo systems disagree. In SQL you would eyeball it. Not any more.
    6. SectionGolden output tests6 minHow you prove a transform still does what it did.
    7. ExerciseSessionize an unsorted event feed35 minin the studioSessions from an unsorted feed — a window function you must build.
    8. QuestionsPython · Conceptual15 minTwenty questions, mostly about what you do when it breaks.

  4. Move 04 · ~3h

    Load only what is new, and survive the re-run

    You can load only new rows, and re-run yesterday without double-counting.

    Skip this if you can already plan a backfill that will not collide tonight.

    1. SectionWhat the order actually decides6 minYou have always lived in the T. Meet E and L.
    2. ExerciseToo slow by morning30 minin the studioThe full refresh stopped finishing. That is how this always starts.
    3. SectionIncremental processing7 minProcess only what is new — and know what "new" means.
    4. ExerciseDeduplicate order snapshots15 minin the studioLatest row per key. The most reused incremental pattern there is.
    5. SectionThe filter that loses a row silently7 minThe watermark that looks correct and quietly drops rows.
    6. Exercisedbt: the incremental branch that loses a row20 minin the studioWrite the filter. Watch the late row vanish, then catch it.
    7. ExerciseRun it for last Tuesday30 minin the studioOne day, six weeks ago, without touching its neighbours.
    8. ExercisePlan an incremental load that loses nothing50 minin the studioPlan the whole load. Lose nothing, double nothing.
    9. QuestionsData Pipeline · Failure & recovery15 minIt broke overnight. Talk through what you do.

  5. Move 05 · ~3h

    Put it on a schedule you can be paged for

    You can schedule dependent work, and say which run produced which rows.

    Skip this if you already know why the logical date is not today’s date.

    1. SectionWhat cron cannot express6 minCron runs things. It has no idea what failed.
    2. ExerciseAfter the first one finishes30 minin the studioDependencies, not timers. The first thing cron cannot do.
    3. SectionWhich runs exist, and when they fire7 minThe run labelled Monday starts on Tuesday. This trips everyone.
    4. ExerciseWhich day does it belong to?30 minin the studioAn event at 23:58 in another timezone. Which partition?
    5. SectionNever ask the clock what day it is6 minThe one habit that makes a backfill possible at all.
    6. ExerciseIt has to be there by six45 minin the studioIt must land by six. The schedule is now a design.
    7. ExerciseSLA-aware alerting flow40 minin the studioAn alert that fires on everything is one nobody reads.
    8. QuestionsAirflow · Operational scenarios15 minThe pager round. You are on call — go.

  6. Move 06 · ~3h

    Make the table a product people depend on

    You can change a table other teams read without breaking them.

    Nothing to skip — this is the move that makes people trust your tables.

    1. SectionThe producer-consumer handshake6 minYou are the producer now. The consumer is you, last year.
    2. ExerciseThe table everyone depends on45 minin the studioForty dashboards read it. Deprecate a column anyway.
    3. SectionA test asks, a constraint promises6 minA test finds out afterwards. A contract refuses to build.
    4. ExerciseIs this change safe?45 minin the studioProve the change is safe before it ships, not after.
    5. SectionShipping breaking changes safely6 minExpand, migrate, contract. Never rename in place.
    6. ExerciseThe payload changed overnight45 minin the studioThe payload changed overnight. Quarantine it, do not crash.
    7. QuestionsData Pipeline · Trade-offs & tool choice15 minThe judgement round: why this one, and what it costs.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Analyst → Data EngineerTracked, 6 moves, ~16hSee the other plans16 more, by what you are aiming at