For complete beginners
Learn data engineering from scratch.
Never built a pipeline? This is the whole climb, in the order a beginner can actually make it: Data Pipeline, Python, SQL, Data Modeling, Airflow, dbt, Apache Spark. You read a little, then build something, at every stage — 8 stages, 37 hands-on exercises, about 19 hours of practice.
Every chapter is free to read without an account. The roadmap tracks itself on your dashboard once you sign in.
- Data Pipeline
- Python
- SQL
- Data Modeling
- Airflow
- dbt
- Apache Spark
01
Who this is for
A data engineer builds the systems that move data from where it is created to where it is used — reliably, every day, without somebody copying spreadsheets by hand. You do not need a computer science degree to start. You need to be comfortable on a computer and curious about why a number is what it is.
- You are starting from zeroA spreadsheet is the most data you have handled. Nothing here assumes more.
- You want the whole mapNot one tool in isolation, but how Python, SQL, modelling, pipelines and Spark fit together.
- You learn by buildingEvery stage after the first has you build something, not only read about it.
02
The roadmap, 8 stages
Each stage buys one capability and uses the ones before it. Skip a stage only if you can already do what it promises.
Stage 01 · ~1h · Data Pipeline
See the whole map before any of the pieces
You can sketch a data platform and say what each box does.
What you will build
- Daily orders ETL
Stage 02 · ~3h · Python
Clean a messy file with Python
You can read records, clean them, validate them and count them.
What you will build
- Count events by type
- Clean spreadsheet headers into column names
- Total revenue by region
- Parse CSV that split(',') cannot
- Deduplicate rows, first one wins
- Validate rows against a schema
Stage 03 · ~3h · SQL
Ask a table a question in SQL
You can filter, group, join and rank rows, and trust the count.
What you will build
- First look at the orders table
- Filter January's bigger orders
- Label orders by value band
- Bucket missing countries
- Revenue by buyer country
- Orders with no line items
- Revenue per customer without double counting
- Paid order CTE decomposition
- Running daily revenue
- First and latest order per buyer
Stage 04 · ~3h · Data Modeling
Design tables that answer questions
You can declare a grain, choose keys, and split facts from dimensions.
What you will build
- Marketplace core entities
- Ride hailing trips
- Social feed and follows
- Cinema seat booking
Stage 05 · ~3h · Data Pipeline
Build a batch pipeline you can re-run
You can build a pipeline that waits, checks itself and survives a rerun.
What you will build
- Your first pipeline
- Waiting for the file
- Counting it once
- Run it for last Tuesday
- Where did the rows go?
- The column that vanished
Stage 06 · ~2h · Airflow · dbt
Schedule it with Airflow, structure it with dbt
You can read an Airflow DAG and lay out a dbt project.
What you will build
- After the first one finishes
- Which day does it belong to?
- dbt: the DAG that never leaves the warehouse
Stage 07 · ~2h · Apache Spark
Run it on Spark when one machine is not enough
You can run a Spark job and explain what it did.
What you will build
- Revenue by country: read the DAG behind one groupBy
- Two counts, two scans: which lines of PySpark actually cost anything?
- Narrow vs wide: what makes Spark cut a new stage?
- Is Spark SQL slower than the DataFrame API? Settle it with explain()
- Why is Parquet faster than CSV? Answer from the plan, not folklore
Stage 08 · ~2h · Data Pipeline
Design one end to end, then say it out loud
You can design a pipeline from requirements and explain every choice.
What you will build
- The history nobody kept
- The spreadsheet is a dependency
03
Where you will practise
SQL and Python have their own practice editors. The bigger ideas get studios of their own — the same three the rest of D8LooP is built around.
- Pipeline StudioDraw the pipeline on a canvas, configure every node, and have the design reviewed.12 exercises in this roadmapTry it — free, no account →
- Modelling StudioTurn a business into tables on an ERD canvas: grain, keys and relationships, reviewed.4 exercises in this roadmapTry it — free, no account →
- Spark PlaygroundRun real PySpark, then read the plan, stages and tasks Spark actually ran.5 exercises in this roadmapBrowse the scenarios →
04
How long it honestly takes
About 19 hours, added up from the steps themselves. That is reference time — how long a step takes someone who already knows it. Budget up to double the first time through: at five hours a week, that is 4 to 8 weeks.
05
When you finish, go here
You can design, build, schedule and explain a data pipeline end to end. From there, the next step is the interview — and each of these plans starts where this one stops.
06
Questions beginners ask first
Can I learn data engineering with no experience?
Yes — this roadmap assumes none. Stage 01 is reading only: what a data platform is and what each part does. Code starts in Stage 02 with small Python exercises, and every stage opens with a short read before it asks you to build anything.
How long does it take to become a data engineer?
Working through this roadmap takes about 19 hours of reference time, and most people should budget up to twice that the first time through — roughly 4 to 8 weeks at five hours a week. That makes you able to build and explain a pipeline end to end. Getting hired usually also means practising for the interview, which is what the plans linked above are for.
Why Python before SQL?
Because cleaning a file of records in Python is the smallest thing a beginner can finish on their own, and it makes the SQL stage easier: GROUP BY is the grouping you already wrote by hand, a set at a time. If you already know some SQL, skip ahead — every stage says what you need to be able to do to skip it.
Do I need to learn Spark as a beginner?
You need to know what it is for, and when you do not need it. The Spark stage is about 2 hours and comes after pipelines, because Spark runs the same transformations you already know, on more than one machine. It runs real PySpark in the Spark Playground, which needs an account. Spark time is limited per day and per month on a free account, so the Spark stage may take a few days to get through; Pro has more.
Is it free?
Every chapter is free to read, with or without an account. All 37 exercises in this roadmap are Foundations or Intermediate, which a free account can attempt. Spark Playground time is limited on a free account. Pro adds more Spark time, more reviews, saved history and the Advanced library.
Do I need to install anything?
No. The chapters, the SQL and Python editors and the three studios all open in the browser. Installing Python, Airflow or dbt yourself is worth doing later; it is not what stops a beginner from learning how the pieces fit.
What should I learn after this?
Interview practice. The plans under "When you finish" start where this roadmap stops: the mid-level loop, a three-week sprint, and the SQL round on its own.
07
Start with stage one
You can sketch a data platform and say what each box does. It takes about 1 hour, and you can read all of it before you make an account.