Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

23.Thesourcethatlies

Stock on hand per SKU per location, pulled hourly from an inventory database on an updated_at cursor. Warehouse staff adjust stock through the app, which maintains that column. Operations correct it in bulk through scripts, which do not — and rows get deleted outright when a line is discontinued. Neither of those has ever appeared in your pipeline.

This pipeline must answer

01Your incremental pull asks what changed. What does the source leave out of that answer?
02How do you find the rows it never mentioned?
03A thousand keys are in your table and gone from the source. Do you delete them?

Guardrails

A storefront reads the published figure to decide what to show as available
The inventory database serves stores during trading hours and cannot absorb a heavy read then
Deleted rows cannot be recovered from the source

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

What runs, and how often — for each of the reads you need?

One source, and more than one cadence.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How fresh does the published stock figure need to be?
e.g.
A storefront decides whether to show something as available. Say what staleness costs there.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Can every read you need run at that cadence?
e.g.
Some reads are cheap and some ask the source for everything. Say what that implies about schedules.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

23.Thesourcethatlies

This pipeline must answer

01Your incremental pull asks what changed. What does the source leave out of that answer?
02How do you find the rows it never mentioned?
03A thousand keys are in your table and gone from the source. Do you delete them?

Guardrails

A storefront reads the published figure to decide what to show as available
The inventory database serves stores during trading hours and cannot absorb a heavy read then
Deleted rows cannot be recovered from the source

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

What runs, and how often — for each of the reads you need?

One source, and more than one cadence.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How fresh does the published stock figure need to be?
e.g.
A storefront decides whether to show something as available. Say what staleness costs there.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Can every read you need run at that cadence?
e.g.
Some reads are cheap and some ask the source for everything. Say what that implies about schedules.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

The source that lies · Pipeline Builder scenario · D8LooP