D8LooP
Start building

Build Your Foundation

Data Engineering Roadmap for Beginners: Zero to Data Engineer

Never built a pipeline? Eight stages, in the order a beginner can actually climb them.

8 stages89 steps~19h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can design, build, schedule and explain a data pipeline end to end.

Data engineer You can build it end to end

02

The plan, 8 stages

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Stage 01 · ~1h

    See the whole map before any of the pieces

    You can sketch a data platform and say what each box does.

    Skip this if you can already explain ingest, store, transform and serve.

    1. SectionLife without a pipeline: the manual tax5 minThe manual work this whole job exists to replace.
    2. SectionAnatomy: four stages, two guardrails5 minThe map for the rest of this plan.
    3. SectionFollow one row: a click becomes a number6 minOne click, followed all the way to a dashboard.
    4. SectionThe five-stage lifecycle6 minEvery later stage of this plan is one of these five.
    5. SectionWhat the order actually decides5 minThe two letters every job posting uses. Know what moved.
    6. SectionLatency: a budget, not a dial5 minWhy you start with batch, and when you would not.
    7. SectionWhich tool goes in which box6 minSo tool names stop being noise and become boxes.
    8. Worked exampleDaily orders ETL30 minin the studioA finished pipeline. Walk it before you build one.

  2. Stage 02 · ~3h

    Clean a messy file with Python

    You can read records, clean them, validate them and count them.

    Skip this if you already write small, tested Python transforms.

    1. SectionThink in records, not syntax4 minThe one habit that separates data code from scripts.
    2. SectionEvery job is input → transform → output5 minEvery exercise in this stage has exactly this shape.
    3. Worked exampleCount events by type20 minin the studioRead the reference answer first, then change it.
    4. ExerciseClean spreadsheet headers into column names25 minin the studioSpreadsheet headers into column names. Your first real cleanup.
    5. SectionDicts: the workhorse record6 minA row is a dict. Most pipeline Python is dicts.
    6. ExerciseTotal revenue by region25 minin the studioGroup and sum by hand, before SQL does it for you.
    7. ExerciseParse CSV that split(',') cannot25 minin the studioWhy split on commas breaks on the first real file.
    8. SectionNulls are not all the same5 minEmpty string, None, "N/A": three different missing values.
    9. ExerciseDeduplicate rows, first one wins20 minin the studioSources send duplicates. Decide which copy wins.
    10. ExerciseValidate rows against a schema25 minin the studioReject bad rows loudly instead of loading them quietly.

  3. Stage 03 · ~3h

    Ask a table a question in SQL

    You can filter, group, join and rank rows, and trust the count.

    Skip this if you already write joins and window functions from memory.

    1. SectionReading a table: rows, columns & grain5 minWhat does one row mean? Ask it of every table.
    2. SectionThe shape of a query4 minThe clauses run in a different order than you write them.
    3. Worked exampleFirst look at the orders table10 minin the studioThe dataset for this stage. Read the reference answer first.
    4. ExerciseFilter January's bigger orders15 minin the studioAND, OR and a date range. Precedence is the trap.
    5. ExerciseLabel orders by value band15 minin the studioCASE turns a number into a label a report can use.
    6. SectionNULL & three-valued logic6 minThe same missing values as Python, with stranger rules.
    7. ExerciseBucket missing countries10 minin the studioMissing countries still count. Give them a bucket.
    8. SectionGROUP BY: one row per group5 minThe Python grouping from stage two, in one clause.
    9. ExerciseRevenue by buyer country15 minin the studioThe shape of almost every report you will ever build.
    10. SectionLEFT JOIN: keep the whole left side5 minThe join that keeps rows with no match.
    11. ExerciseOrders with no line items12 minin the studioFind what is missing, the first audit every engineer runs.
    12. SectionFan-out: when a join doubles your metric6 minWhy a correct-looking total can be twice too big.
    13. ExerciseRevenue per customer without double counting15 minin the studioYour total will come out too big. That is the lesson.
    14. ExercisePaid order CTE decomposition15 minin the studioStretch: CTEs, so a long query reads top to bottom.
    15. SectionOVER: partition, order & frame6 minA window keeps every row and still sees its neighbours.
    16. ExerciseRunning daily revenue15 minin the studioStretch: a running total, the friendliest window there is.
    17. ExerciseFirst and latest order per buyer15 minin the studioStretch: first and latest per key. You will reuse it weekly.

  4. Stage 04 · ~3h

    Design tables that answer questions

    You can declare a grain, choose keys, and split facts from dimensions.

    Skip this if you can already draw a star schema and defend its grain.

    1. SectionWhat a data model actually is4 minUp to now you queried tables. Now you decide them.
    2. SectionEntities & attributes5 minFind the nouns in a business, and what describes them.
    3. Worked exampleMarketplace core entities25 minin the studioThe same marketplace you queried, modelled. Study every answer.
    4. SectionDeclaring grain before you build6 minThe fan-out you hit in SQL, prevented at design time.
    5. SectionPrimary keys & candidate keys5 minWhat makes one row different from every other row.
    6. ExerciseRide hailing trips30 minin the studioOne person, two roles. Model it without duplicating people.
    7. SectionMany-to-many bridges6 minThe relationship a single foreign key cannot hold.
    8. ExerciseSocial feed and follows30 minin the studioA follow points at a user on both ends.
    9. SectionFact tables: what you measure6 minAnalytics tables have a different shape. Facts come first.
    10. SectionThe star schema in action6 minFacts in the middle, dimensions around. Most warehouses look like this.
    11. ExerciseCinema seat booking30 minin the studioGive the right thing its own row and double-booking disappears.

  5. Stage 05 · ~3h

    Build a batch pipeline you can re-run

    You can build a pipeline that waits, checks itself and survives a rerun.

    Skip this if your pipelines are already idempotent and validated.

    1. SectionClean, shape, aggregate5 minEverything you did in Python and SQL, named as pipeline steps.
    2. ExerciseYour first pipeline30 minin the studioOne file a night, one report by morning.
    3. ExerciseWaiting for the file30 minin the studioStop betting that somebody else’s file arrived on time.
    4. SectionIdempotency: safe to repeat5 minEvery pipeline runs twice eventually. Plan for it now.
    5. ExerciseCounting it once30 minin the studioThe rerun that doubles revenue, and the design that cannot.
    6. SectionPartitions: the unit of reprocessing6 minProcess one day at a time, so one bad day is fixable.
    7. ExerciseRun it for last Tuesday30 minin the studioRe-run one past day without touching the rest.
    8. SectionWriting validation checks6 minThe Python validation from stage two, as a pipeline gate.
    9. ExerciseWhere did the rows go?30 minin the studioRows vanished between source and report. Make it impossible silently.
    10. ExerciseThe column that vanished30 minin the studioThe source changed shape overnight. Catch it before the dashboard.

  6. Stage 06 · ~2h

    Schedule it with Airflow, structure it with dbt

    You can read an Airflow DAG and lay out a dbt project.

    Skip this if you have shipped an Airflow DAG and a dbt model.

    1. SectionWhen one script becomes a system5 minThe moment your pipeline needs something to run it.
    2. SectionWhy the graph must not loop5 minDAG is the word Airflow, dbt and Spark all share.
    3. SectionThe whole file, and why each line is there7 minA real DAG file, read line by line.
    4. ExerciseAfter the first one finishes30 minin the studioReplace "wait thirty minutes" with a real dependency.
    5. SectionWhich runs exist, and when they fire7 minWhy Monday’s run fires on Tuesday. Everyone trips once.
    6. ExerciseWhich day does it belong to?30 minin the studioLate events land on the wrong day unless you decide.
    7. SectionWhat dbt actually does5 minYour SQL, versioned, tested and built in the right order.
    8. SectionYour first model6 minA model is a SELECT in a file. That is all.
    9. Sectionref(): the one call that builds the graph6 minOne function, and dbt works out the build order.
    10. Exercisedbt: the DAG that never leaves the warehouse30 minin the studioLay out staging and marts, the way dbt projects do.
    11. SectionWhy your SQL moved into a repo5 mindbt lives in Git. So does every team you will join.

  7. Stage 07 · ~2h

    Run it on Spark when one machine is not enough

    You can run a Spark job and explain what it did.

    Skip this if you can already read a Spark plan and its stages.

    1. SectionWhen one machine isn’t enough5 minThe same transforms, once the data outgrows a laptop.
    2. SectionWhen to use Spark, and when not to5 minMost data does not need Spark. Know the line.
    3. SectionDriver & executors: the brain and the workers6 minWho plans the work, and who does it.
    4. SectionLazy evaluation: nothing runs until it must6 minThe line that looks slow is rarely the slow one.
    5. Worked exampleRevenue by country: read the DAG behind one groupBy20 minin the studioYour SQL stage’s revenue report, run on real Spark.
    6. ExerciseTwo counts, two scans: which lines of PySpark actually cost anything?10 minin the studioTwo counts, two scans. Prove it from the job list.
    7. ExerciseNarrow vs wide: what makes Spark cut a new stage?15 minin the studioWhy a groupBy splits the job and a filter does not.
    8. ExerciseIs Spark SQL slower than the DataFrame API? Settle it with explain()15 minin the studioYour SQL carries over. Same plan either way.
    9. ExerciseWhy is Parquet faster than CSV? Answer from the plan, not folklore20 minin the studioWhy data teams store Parquet, answered from the plan.
    10. SectionReading the Spark UI: a tab-by-tab tour7 minWhere you look first when a job is slow.

  8. Stage 08 · ~2h

    Design one end to end, then say it out loud

    You can design a pipeline from requirements and explain every choice.

    Nothing to skip — this is where the earlier stages become one skill.

    1. SectionThe requirements-first method6 minAsk what is needed before you pick a single tool.
    2. SectionThe questions to ask first6 minThe questions that shape a design before any box appears.
    3. ExerciseThe history nobody kept30 minin the studioCapstone: a pipeline and a model, designed together.
    4. ExerciseThe spreadsheet is a dependency30 minin the studioTwo inputs, one edited by hand. Design for both.
    5. SectionNarrating a design6 minExplaining a design is its own skill. Practise it here.
    6. InterviewOne letter moved. What went with it?6 minStage one’s map, explained without notes.
    7. InterviewWhat is the grain of a table6 minAsked in nearly every modelling round. Answer in one sentence.
    8. InterviewFact or dimension6 minStage four, out loud: measured or described?
    9. InterviewThe four idempotent writes6 minStage five in one answer: why a rerun is safe.
    10. InterviewYou already have Airflow6 minStage six: what each tool is for, and why both.
    11. InterviewThe three components, named6 minStage seven: the opening question of most Spark screens.
    12. QuestionsData Pipeline · Conceptual15 minTwenty questions across the whole plan. Say, then check.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Zero to Data EngineerTracked, 8 stages, ~19hSee the other plans16 more, by what you are aiming at