Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

extracttransformloadorchestrate

Scenario

37.Tenterabytesofclicks,andaquestionaboutsessions

A streaming service captures every tap from web, iOS, Android and three TV apps — four hundred million events a day, about ten terabytes. Product wants to know how long people spend in the app before they start something and where they drop out. Design the pipeline that turns an endless stream of clicks into sessions.

This pipeline must answer

01How long did a viewer spend in the app before they started something, and where did they drop off?
02A phone was in a tunnel and uploads forty minutes of events at once. Which session do they belong to?
03Somebody redefines a session from a 30-minute gap to 15. What has to be rebuilt, and how far back?

Guardrails

A session is defined by a gap in a person’s behaviour, not by a clock
Devices that lose signal upload their backlog when they reconnect
Yesterday’s report is read at 09:00 and must be final
The definition of a session will change and the last ninety days will have to be rebuilt
Include a check before anything is published, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Four hundred million events a day, and product wants yesterday’s session report by 09:00. What makes work happen?

Say what runs, and how often — a report by 09:00 and a session that is still open are two different clocks.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Is anything always running, is anything scheduled, or both?
e.g.
Name each thing that produces work and what it is triggered by. If the answer is “both”, say what each half is responsible for.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Which of the two decides whether the 09:00 report exists?
e.g.
Point at one of them and say what happens to the report if it does not run.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Preparing your workspaceloading the brief, studio & editor…

extracttransformloadorchestrate

Scenario

37.Tenterabytesofclicks,andaquestionaboutsessions

This pipeline must answer

01How long did a viewer spend in the app before they started something, and where did they drop off?
02A phone was in a tunnel and uploads forty minutes of events at once. Which session do they belong to?
03Somebody redefines a session from a 30-minute gap to 15. What has to be rebuilt, and how far back?

Guardrails

A session is defined by a gap in a person’s behaviour, not by a clock
Devices that lose signal upload their backlog when they reconnect
Yesterday’s report is read at 09:00 and must be final
The definition of a session will change and the last ninety days will have to be rebuilt
Include a check before anything is published, and a failure path
Avoid invalid cycles

Act 1 · Specify — answer all 9 and your spec flows0 / 9 answered

1Trigger & cadence

Four hundred million events a day, and product wants yesterday’s session report by 09:00. What makes work happen?

Say what runs, and how often — a report by 09:00 and a session that is still open are two different clocks.

This choice changes which questions you are asked next — and the pipeline you end up drawing.

◆ What you are committing to on the canvas

Is anything always running, is anything scheduled, or both?
e.g.
Name each thing that produces work and what it is triggered by. If the answer is “both”, say what each half is responsible for.
- configTrigger `Runs` = daily / hourly — leaving it unset is what fails the freshness check
Which of the two decides whether the 09:00 report exists?
e.g.
Point at one of them and say what happens to the report if it does not run.
- nodeOrchestration & contracts → Scheduler trigger for a clock; Amazon MWAA, Step Functions, Dagster or Prefect when one step has to wait for another to finish

New to this?Batch vs Streaming vs Micro-batch

Edit your answer here✓ captured

Markdown: **bold** · *italic* · # heading · - bullet

Act 2 · Architect

Practice studio

Ctrl/Cmd + C copyCtrl/Cmd + V pasteCtrl/Cmd + Z undoDelete remove

40%

Design review

The reviewer reads your pipeline.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules

Ten terabytes of clicks, and a question about sessions · Pipeline Builder scenario · D8LooP