Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

9.Fourteenmonthsofwrongnumbers

GridMeter is a utility data platform. Two million smart meters send a reading every fifteen minutes, and a nightly pipeline turns them into the billing intervals customer care and billing read each morning. You have just found that a conversion has been wrong for fourteen months. Every billing interval in that period has to be recomputed. Design the pipeline, and design how that reprocess actually runs.

This pipeline must answer

01Fourteen months of billing intervals are wrong. How does the reprocess actually run?
02It dies forty hours in, at three in the morning. What happens next?
03Tonight's run still has to make its deadline. What stops the backfill breaking it?

Guardrails

Tonight’s run still has to make its deadline
The warehouse is serving billing and customer care throughout
The reprocess has to survive being killed partway through
Every rebuilt day is checked the same as a fresh one
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

There are two things that need to run here. What starts each of them?

The nightly deadline, and the fourteen months — and why one must never be the other.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

What is the nightly job's deadline, and who set it?
e.g.
Billing reads the previous day each morning, so the run has to be finished before then. That deadline exists whether or not you are reprocessing anything.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Should the reprocess be a flag on that job, or its own thing?
e.g.
A flag is one line of code and it puts a fourteen-month job on the same schedule, the same retries and the same alerting as a job with a deadline tonight.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

9.Fourteenmonthsofwrongnumbers

This pipeline must answer

01Fourteen months of billing intervals are wrong. How does the reprocess actually run?
02It dies forty hours in, at three in the morning. What happens next?
03Tonight's run still has to make its deadline. What stops the backfill breaking it?

Guardrails

Tonight’s run still has to make its deadline
The warehouse is serving billing and customer care throughout
The reprocess has to survive being killed partway through
Every rebuilt day is checked the same as a fresh one
Avoid invalid cycles

Act 1 · Specify — answer all 8 and your spec flows0 / 8 answered

1Trigger & cadence

There are two things that need to run here. What starts each of them?

The nightly deadline, and the fourteen months — and why one must never be the other.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

What is the nightly job's deadline, and who set it?
e.g.
Billing reads the previous day each morning, so the run has to be finished before then. That deadline exists whether or not you are reprocessing anything.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Should the reprocess be a flag on that job, or its own thing?
e.g.
A flag is one line of code and it puts a fourteen-month job on the same schedule, the same retries and the same alerting as a job with a deadline tonight.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Fourteen months of wrong numbers · Pipeline Builder scenario · D8LooP