Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

11.Itworks,anditcoststoomuch

SensorCo monitors industrial machines. Devices report every few seconds; ops watch a wall-mounted dashboard refreshing every five seconds; analysts want daily averages across years of history. The pipeline is correct and it costs £40,000 a month. Nothing is broken. Design it, and work out what it is paying for that nobody asked for.

This pipeline must answer

01What is every machine reading right now — on a wall dashboard refreshing every five seconds?
02What were the daily averages for this machine across the last three years?
03Why does this cost £40,000 a month, and what would you cut first?

Guardrails

The five-second dashboard is a hard requirement with a real consequence behind it
Analysts need years of daily history
A summary must remain rebuildable for long enough to fix a mistake
Include a check before anything is served, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

A wall dashboard refreshes every five seconds. Analysts want three years of daily averages. What runs, and how often?

Two consumers at wildly different speeds — and the cadence question is a cost question here.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Which consumer sets the fastest requirement, and what is it?
e.g.
The ops wall board. Five seconds is not a preference — a threshold crossing that arrives a minute late is a machine that ran a minute too long.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Does the slow half need to keep up with the fast half?
e.g.
No, and assuming it does is one of the more expensive mistakes available. Ask what the analysts actually open, and how often.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

11.Itworks,anditcoststoomuch

This pipeline must answer

01What is every machine reading right now — on a wall dashboard refreshing every five seconds?
02What were the daily averages for this machine across the last three years?
03Why does this cost £40,000 a month, and what would you cut first?

Guardrails

The five-second dashboard is a hard requirement with a real consequence behind it
Analysts need years of daily history
A summary must remain rebuildable for long enough to fix a mistake
Include a check before anything is served, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

A wall dashboard refreshes every five seconds. Analysts want three years of daily averages. What runs, and how often?

Two consumers at wildly different speeds — and the cadence question is a cost question here.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Which consumer sets the fastest requirement, and what is it?
e.g.
The ops wall board. Five seconds is not a preference — a threshold crossing that arrives a minute late is a machine that ran a minute too long.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Does the slow half need to keep up with the fast half?
e.g.
No, and assuming it does is one of the more expensive mistakes available. Ask what the analysts actually open, and how often.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

It works, and it costs too much · Pipeline Builder scenario · D8LooP