Loadingone moment…
Pipeline Builder · Practice Studio
A visual DAG studio for the orchestration questions you'll actually be asked — pick a scenario, build the pipeline on a live canvas, then defend it.
Get clear on lifecycle, ordering, retries, and quality first — 24 chapters that make every scenario below click.
Open the Data Pipeline knowledge baseCompose a daily ingestion pipeline with validation, transform, and warehouse load steps.
One of the two inputs is a spreadsheet somebody edits by hand, and it decides what the pipeline believes should have happened.
One file a night, one report by morning.
The file usually lands at 01:00.
The warehouse has fewer trips than the app does, nothing has ever failed, and it has been like this for months.
The producer retries, so the same event arrives twice, and follower counts are visibly too high.
The job took twenty minutes when it was written and takes six hours now.
Another team renamed a column on a Tuesday afternoon.
Every booking carries three timestamps and the report says "bookings yesterday".
Seven per cent of matches never reach the report and every row involved is perfectly valid.
The second job runs thirty minutes after the first, because the first usually takes twenty.
The HR system only knows about right now, somebody asked about January, and there is no answer anywhere.
The job works out yesterday’s date for itself, so it can only ever produce yesterday.
Four hundred lines of SQL somebody runs by hand, over tables another team already loaded.
Model a pipeline that retries safely and escalates when freshness SLAs are missed.
Millions of records a day from a marketplace app — orders, payments, listings, shipments.
Parcel scans made on handhelds and drivers’ phones arrive late and out of order.
A category dashboard that loads in milliseconds and gives the same answer for last March however many times merchandising has renamed the tree since — from a source that only ever shows you today.
Three years of daily rates from a vendor API that allows sixty calls a minute, bills per call, and returns a 429 at page 40 of 90.
The pipeline already runs and finance reads it every morning.
You own the MRR table and twelve other teams build on it.
Loads arrive every fifteen minutes; the carriers they name arrive once a night.
The models decide a figure a regulator reads, and you are not allowed a copy of the data they run on.
Analysts need the warehouse within five minutes of a billing database another team owns — including the orders that get deleted.
Coordinate multiple source streams into one downstream mart with guardrails.
Design a live events pipeline — Kinesis → Flink → S3 (bronze/gold) → Athena, plus a real-time serving index — with exactly-once processing and a dead-letter path.
A weekly run that sends money to creators’ banks.
A conversion has been wrong for fourteen months and two million meters have to be reprocessed — without breaking tonight’s run, saturating the warehouse everyone else is using, or leaving history half-corrected.
Events arrive from client apps another team ships on their own schedule.
Nothing is broken and it costs £40,000 a month.
A bid disqualified four days late changes who won an auction you have already invoiced.
A person asks to be deleted, and the pipeline has written them to a raw landing, a warehouse profile and a serving store.
Dispatchers assign the day’s work at 06:00 and there is more overnight work than obviously fits.
Four hundred customers on shared infrastructure, and the largest sends more in an hour than the smallest sends in a year.
A pricing pipeline that has been right for six years is moving to a new platform, and the old one switches off on the 31st.
The source keeps a timestamp of when it last changed a row, and it is not always right.
Sixty organisations submit on their own schedules, the publication date is in legislation, and eleven of them have not sent anything.
Four hundred million events a day from six client platforms, and product wants to know how long people spend in the app before they start something.
A pricing service asks for a multiplier thousands of times a second and charges people whatever it is told.
An annual year-in-review campaign launches at 09:00 and traffic reaches ten times normal within four minutes — and stays elevated for a fortnight.