Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

5.Marketplacetransactionsatscale

A marketplace app is used by hundreds of thousands of buyers and sellers. Every order, payment, listing edit and shipment update produces a record — millions a day, and growing. None of it is stored for analysis yet, and the answers are read each morning. Design the pipeline that turns that into something the business can answer questions from.

This pipeline must answer

01What did the marketplace sell yesterday, by category?
02Which sellers are growing, and which are stalling?
03A bug in the cleaning is found a month later — can every number be rebuilt?

Guardrails

Nothing you build may slow the app down
A bug in the cleaning found a month later has to be repairable — every number back to then must be rebuildable
Include a check before anything the business reads, and somewhere for records you cannot use
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Records arrive all day. The answers are read each morning. So what runs, and when?

There are two cadences hiding in that sentence. Find them both before you pick anything.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Can capture wait until tonight?
e.g.
No — a record the app emitted at 14:00 has to be somewhere by 14:00, or it is simply gone. Capture cannot be scheduled.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Does the reshaping have to keep up with the capture?
e.g.
The answers are read the next morning, so the expensive work runs once over a closed day rather than on every record as it lands.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

5.Marketplacetransactionsatscale

This pipeline must answer

01What did the marketplace sell yesterday, by category?
02Which sellers are growing, and which are stalling?
03A bug in the cleaning is found a month later — can every number be rebuilt?

Guardrails

Nothing you build may slow the app down
A bug in the cleaning found a month later has to be repairable — every number back to then must be rebuildable
Include a check before anything the business reads, and somewhere for records you cannot use
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Records arrive all day. The answers are read each morning. So what runs, and when?

There are two cadences hiding in that sentence. Find them both before you pick anything.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Can capture wait until tonight?
e.g.
No — a record the app emitted at 14:00 has to be somewhere by 14:00, or it is simply gone. Capture cannot be scheduled.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Does the reshaping have to keep up with the capture?
e.g.
The answers are read the next morning, so the expensive work runs once over a closed day rather than on every record as it lands.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Marketplace transactions at scale · Pipeline Builder scenario · D8LooP