Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

25.Yourfirstpipeline

Parking sensors across a city report when a bay becomes occupied and when it frees up. One CSV lands overnight from the vendor. By 08:00 the enforcement team needs to know how long each street was occupied yesterday, so they can decide where to send wardens today.

This pipeline must answer

01How long was each street occupied yesterday?
02The sensor file has not arrived by 03:00. What happens?
03The job runs twice by accident. What does the report say?

Guardrails

The file usually arrives by 02:00, sometimes late, occasionally not at all
The report is acted on within hours of appearing
Nothing here is large enough to need a distributed engine

Act 1 · Specify — answer all 7 and your spec flows0 / 7 answered

1Trigger & cadence

What starts this off, and how often?

The first thing that makes it a pipeline rather than something somebody remembers to run.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How often does the report need to be produced?
e.g.
Somebody reads it at 08:00 to decide where enforcement goes today. Say what cadence that implies.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
What time would you pick, and what are you between?
e.g.
There is a thing that must have happened before, and a thing that happens after. Name both.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

25.Yourfirstpipeline

This pipeline must answer

01How long was each street occupied yesterday?
02The sensor file has not arrived by 03:00. What happens?
03The job runs twice by accident. What does the report say?

Guardrails

The file usually arrives by 02:00, sometimes late, occasionally not at all
The report is acted on within hours of appearing
Nothing here is large enough to need a distributed engine

Act 1 · Specify — answer all 7 and your spec flows0 / 7 answered

1Trigger & cadence

What starts this off, and how often?

The first thing that makes it a pipeline rather than something somebody remembers to run.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How often does the report need to be produced?
e.g.
Somebody reads it at 08:00 to decide where enforcement goes today. Say what cadence that implies.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
What time would you pick, and what are you between?
e.g.
There is a thing that must have happened before, and a thing that happens after. Name both.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Your first pipeline · Pipeline Builder scenario · D8LooP