extracttransformloadorchestrate
Scenario
2.SLA-awarealertingflow
Design an hourly pipeline for a team that depends on fresh data by a deadline: retry transient failures safely, monitor freshness, and escalate to the on-call the moment the SLA is at risk or missed — never fail silently.
This pipeline must answer
- 01Is the data fresh by its deadline — and who is paged the moment it isn’t?
- 02When a step fails transiently, does a retry fix it without double-counting?
- 03Can we see the pipeline’s health before a consumer complains?
Guardrails
- Hourly freshness SLA
- Retries must be idempotent
- Include an observability/monitor and an escalation (alert) path, plus a dead-letter path
- Avoid invalid cycles
Act 1 · Ask — answer all 11 and your spec flows11 / 11 answered
1Consumer & goal
Who reads this data, and what do they decide with it?
Name a real team, the decision they make, and the time they need it by.
- Who opens this first thing in the morning?
e.g.
The finance analyst who signs off yesterday’s revenue — a job title, not “the business”. - What decision changes because of this number?
e.g.
Reorder stock, chase a refund spike, close the books for the day. - What goes wrong if the number is wrong?
e.g.
A published revenue figure gets restated. That sets the bar: slightly late is survivable, wrong is not. - By when do they need it?
e.g.
“On my screen by 9am” is a deadline you can design against. “Fresh” is not.
◈ What this decides
New to this?What a Data Pipeline Is and Why It MattersThe output shape, how correct it must be, and how loudly it may fail.
Markdown: **bold** · *italic* · # heading · - bullet
signals:
✓ answered — flow continues through this valveAct 2 · Architect
Design review
The reviewer reads your pipeline.
Run the review to check your architecture against this scenario — a valid DAG, a trigger for the freshness SLA, a transform stage, a quality gate before publish, and a failure path — then defend it in a short interview.