Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

36.Fiveminutesbehindthesource

Marlow’s billing database is a Postgres owned by the payments team: 4.2 million orders, updated all day, and rows are genuinely deleted. Analysts and a customer-support tool need the warehouse within five minutes, including the orders that disappear. Design the pipeline that reads changes rather than rows.

This pipeline must answer

01An order is deleted in the billing database. What does the warehouse do about it, and when?
02Two changes to the same order arrive in the wrong order. Which value ends up in the table?
03The connector has been down since Friday evening. What is the first thing you check on Monday?

Guardrails

You do not own the source and cannot add load to it
A deleted order must be answerable afterwards, not merely absent
Two changes to the same order can reach you in either order
Include a check before anything is published, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Analysts want the warehouse within five minutes of the billing database. What starts a run?

Five minutes is a deadline, not a cadence. Say what makes work happen and how you would know it had stopped.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Is there a run at all, or is something always running?
e.g.
Name the thing that produces work. If it is a schedule, say the interval; if it is continuous, say what it is consuming and what happens when that goes quiet.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Five minutes of what — of the change happening, or of you hearing about it?
e.g.
State where you measure from and where you measure to. A budget you cannot point at two clocks for is not a budget.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

36.Fiveminutesbehindthesource

This pipeline must answer

01An order is deleted in the billing database. What does the warehouse do about it, and when?
02Two changes to the same order arrive in the wrong order. Which value ends up in the table?
03The connector has been down since Friday evening. What is the first thing you check on Monday?

Guardrails

You do not own the source and cannot add load to it
A deleted order must be answerable afterwards, not merely absent
Two changes to the same order can reach you in either order
Include a check before anything is published, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Analysts want the warehouse within five minutes of the billing database. What starts a run?

Five minutes is a deadline, not a cadence. Say what makes work happen and how you would know it had stopped.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Is there a run at all, or is something always running?
e.g.
Name the thing that produces work. If it is a schedule, say the interval; if it is continuous, say what it is consuming and what happens when that goes quiet.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Five minutes of what — of the change happening, or of you hearing about it?
e.g.
State where you measure from and where you measure to. A budget you cannot point at two clocks for is not a budget.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Five minutes behind the source · Pipeline Builder scenario · D8LooP