extracttransformloadorchestrate
Scenario
4.Realtimeeventspipeline
Design a live events pipeline for a product emitting clickstream/telemetry: ingest events off a streaming bus, process them in flight, land raw then curated data in a layered lake, and serve both a real-time index and ad-hoc analytics — at sub-minute freshness, with a reprocessable raw history in S3.
This pipeline must answer
- 01What is happening right now — events available within seconds, not hours?
- 02Can we replay a failed window without double-counting?
- 03Which events failed, and are they quarantined off the live path?
Guardrails
- Sub-minute (streaming) SLA
- Land raw as bronze and serve curated as gold
- Processing must be exactly-once
- Include a dead-letter path
- Avoid invalid cycles
Act 1 · Ask — answer all 11 and your spec flows11 / 11 answered
1Consumer & goal
Who reads this data, and what do they decide with it?
Name a real team, the decision they make, and the time they need it by.
- Who opens this first thing in the morning?
e.g.
The finance analyst who signs off yesterday’s revenue — a job title, not “the business”. - What decision changes because of this number?
e.g.
Reorder stock, chase a refund spike, close the books for the day. - What goes wrong if the number is wrong?
e.g.
A published revenue figure gets restated. That sets the bar: slightly late is survivable, wrong is not. - By when do they need it?
e.g.
“On my screen by 9am” is a deadline you can design against. “Fresh” is not.
◈ What this decides
New to this?What a Data Pipeline Is and Why It MattersThe output shape, how correct it must be, and how loudly it may fail.
Markdown: **bold** · *italic* · # heading · - bullet
signals:
✓ answered — flow continues through this valveAct 2 · Architect
Design review
The reviewer reads your pipeline.
Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.