D8LooP
Start building

Hands on Concepts

Owning the Data Engineering On-Call Pager

One incident, in four phases: notice it, diagnose it, repair it, prevent it.

4 phases36 steps~9h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can take the pager and give the numbers back without a second incident.

Pager handed over You can be trusted with the pager

02

The plan, 4 phases

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Phase 01 · ~2h

    Find out before they do

    Your monitoring tells you first, and only for things worth waking for.

    Skip this if freshness and volume alerts already page you before anyone else.

    1. SectionA pipeline failing without failing7 minGreen job, wrong numbers. The failure mode nothing catches by default.
    2. SectionThe five pillars of observability6 minFreshness, volume, schema, distribution, lineage. Check all five.
    3. SectionSLIs, SLOs & alerts people answer7 minAn alert nobody answers is not monitoring. It is a muted channel.
    4. SectionThe failures that never raise6 minThe task succeeded and wrote nothing. Airflow will not tell you.
    5. SectionFreshness: one subtraction6 minStale source, healthy pipeline. Check the input, not just the job.
    6. ExerciseSLA-aware alerting flow40 minin the studioWire an alert to an SLA rather than to a task state.
    7. InterviewWrong numbers, one hour to the meeting6 minShip a labelled fallback while you fix. The move people skip.
    8. QuestionsAirflow · Operational scenarios15 minWhat to alert on, and the runbook you wish you had written.

  2. Phase 02 · ~2h

    Diagnose without guessing

    You narrow a wrong number to one layer before changing anything.

    Skip this if you have a triage order you follow instead of a hunch.

    1. SectionA triage order that works7 minAn order you follow beats a hunch at three in the morning.
    2. SectionDebugging a wrong number7 minBisect the pipeline by row count. Cheaper than reading code.
    3. SectionReconciliation: do the numbers agree?6 minSource against target, per key. Matching totals prove nothing.
    4. SectionSymptom to diagnosis6 minA named symptom maps to a few causes. Learn the map.
    5. SectionLineage: cause and blast radius6 minWho else is wrong right now? Answer that before you fix anything.
    6. ExerciseData-quality audit summary20 minin the studioNulls, duplicates, orphans, freshness — in one query you can rerun.
    7. ExerciseReferential integrity audit15 minin the studioOrphan keys are where a silent drop usually turns out to be.
    8. ExerciseWhere does skew show up? Reading task metrics, not the DAG20 minin the studioA Spark job stuck on one task: prove skew before blaming it.
    9. InterviewTwo P1s at the same time6 minRank on reversibility. The running job outranks the stopped one.
    10. QuestionsAirflow · Debug & fix15 minStuck, zombied, green-but-wrong. Twenty real diagnoses.

  3. Phase 03 · ~3h

    Repair without making it worse

    You can rerun and backfill without double-counting anything.

    Skip this if you already know which of your loads are safe to rerun.

    1. SectionFour ways to write safely7 minBefore you rerun anything, know which pattern the task uses.
    2. SectionAtomic publishing6 minWrite aside, swap at the end. Nobody reads a half-finished table.
    3. InterviewAppend, merge or overwrite6 minAppend duplicates on a rerun. The whole risk, in one line.
    4. SectionChoosing the right repair7 minClear, mark success, or rerun. Three repairs, three different results.
    5. SectionA bug shipped eleven days ago7 minHistory is wrong and the pipeline is still running. Both at once.
    6. SectionRecovery: retries & checkpoints6 minRestart from the last safe point, not from the beginning.
    7. ExerciseFourteen months of wrong numbers50 minin the studioFourteen months of wrong numbers, and production still running.
    8. ExerciseYou fixed it. Who already used it?50 minin the studioYou fixed it. Who already used the wrong numbers?
    9. QuestionsData Pipeline · Failure & recovery15 minTwenty reruns and replays, each with the target state after.

  4. Phase 04 · ~3h

    Make it not happen again

    The check that would have caught it exists before you go back to bed.

    Nothing to skip — an incident with no new check will repeat.

    1. SectionWhere the check lives in the pipeline6 minA check after the load tells you late. Before stops it.
    2. SectionQuarantine & circuit breakers7 minDecide in advance what stops the pipeline and what only warns.
    3. InterviewFail the record or fail the job?6 minA declared threshold, written down before the night it matters.
    4. SectionWhat errors, what warns, what skips6 minA warning nobody reads is the same as no test at all.
    5. SectionAlerting without fatigue6 minEvery noisy alert makes the real one less likely seen.
    6. SectionRunning the incident & writing it up7 minThe write-up is the only part that changes what happens next time.
    7. ExerciseIs this change safe?45 minin the studioJudge a change before it ships. That is where incidents start.
    8. ExerciseThe source that lies60 minin the studioThe upstream is confidently wrong and nothing errors. Catch it anyway.
    9. QuestionsData Pipeline · Trade-offs & tool choice15 minBlock or warn, and who owns it. The post-incident arguments.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Own the on-call pagerTracked, 4 phases, ~9hSee the other plans16 more, by what you are aiming at