Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

13.Thesourcewillnotletyou

EnergyTrade books trades in several currencies and reports them in a base currency using the exchange rate as of the trade date. The rates come from a vendor API that allows sixty calls a minute, returns a hundred rates per page, bills per call, and refuses rather than queues. You need three years of it. Design the pipeline.

This pipeline must answer

01You need three years of daily rates from an API that allows sixty calls a minute. How?
02It returns 429 at page 40 of 90. Where do you start again from?
03The vendor revises a rate published last week. What happens to the reports that used it?

Guardrails

The vendor sets the pace and you cannot negotiate with it
Every call is billed
It revises published rates days later
An extract that stops halfway must not publish silently
Include a check before publish and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

Rates publish once a day. Trades book continuously. What runs, and how often?

The cadence — and why running more often is worse than useless here.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How often does the thing you are pulling actually change?
e.g.
The vendor publishes daily. Polling hourly gets you the same numbers twenty-four times and bills you for each one.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Is one pull a day enough, given revisions?
e.g.
Not quite. The vendor sometimes revises a rate days later, so the run has to look back a little as well as forward.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

13.Thesourcewillnotletyou

This pipeline must answer

01You need three years of daily rates from an API that allows sixty calls a minute. How?
02It returns 429 at page 40 of 90. Where do you start again from?
03The vendor revises a rate published last week. What happens to the reports that used it?

Guardrails

The vendor sets the pace and you cannot negotiate with it
Every call is billed
It revises published rates days later
An extract that stops halfway must not publish silently
Include a check before publish and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

Rates publish once a day. Trades book continuously. What runs, and how often?

The cadence — and why running more often is worse than useless here.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

How often does the thing you are pulling actually change?
e.g.
The vendor publishes daily. Polling hourly gets you the same numbers twenty-four times and bills you for each one.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Is one pull a day enough, given revisions?
e.g.
Not quite. The vendor sometimes revises a rate days later, so the run has to look back a little as well as forward.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

The source will not let you · Pipeline Builder scenario · D8LooP