D8LooP
Build my plan

Interviewing

How to Prepare for the Data Pipeline Design Round

The diagram is the easy half. This is the half that gets people rejected.

4 parts39 steps~12h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can design a pipeline and defend the reruns, the late data and the cost.

Design day You can defend a design under questioning

02

The plan, 4 parts

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Part 01 · ~2h

    Requirements before boxes

    You ask what the pipeline is for before choosing anything to build it.

    Skip this if you ask about latency, volume and consumers unprompted.

    1. SectionThe eight steps7 minNaming tools before requirements is the documented top elimination.
    2. SectionStep 1: ask questions6 minLatency, volume, consumers, retention. Ask before you draw.
    3. SectionSteps 2 and 3: capture, speed6 minBudget, team size, what already exists. These decide more than tech.
    4. SectionLatency, cost and effort6 minTurn "real time" into a number before you agree to build it.
    5. SectionAnswer the five-minute prompt6 minChoosing streaming when batch would do is a real mark against you.
    6. InterviewChoosing a partition key6 minOne choice sets ordering, distribution, hotspots and scale. Say all four.
    7. ExerciseRealtime events pipeline45 minin the studioA prompt lifted from real loops: batch or streaming, and why.
    8. InterviewWhat ordering a stream gives you6 minPer partition, never global. A flat "yes" designs a reordering bug.
    9. QuestionsData Pipeline · Conceptual15 minThe vocabulary the rest of the round is conducted in.

  2. Part 02 · ~4h

    Make it safe to run twice

    You can say what happens on a rerun without checking the code.

    Skip this if every load you design is a merge or a partition overwrite.

    1. SectionAt-most, at-least, exactly-once6 minThree guarantees, three costs. Pick one and say what it buys.
    2. SectionIdempotency: safe to repeat7 minThe word rejection notes use when it is missing. Say it early.
    3. SectionDeduplication & idempotency keys6 minWho assigns the key decides whether dedup works at all.
    4. SectionAppend, upsert, overwrite6 minThree write modes, and only one of them duplicates on a rerun.
    5. InterviewAppend, merge or overwrite6 minRun it twice. Append duplicates; the other two converge.
    6. ExerciseThe fact arrived first45 minin the studioThe fact beat its dimension. What does your load do about it?
    7. ExerciseYou fixed it. Who already used it?50 minin the studioYou fixed it and replayed. Who already used the wrong numbers?
    8. InterviewDeduplicating with a version6 minDedup handles copies. Only a version handles arriving out of order.
    9. StepPipeline round, halfway: safe to run twice78 minin the studioHalfway check: two canvases, three rerun questions.
    10. QuestionsData Pipeline · Failure & recovery15 minTwenty reruns, replays and backfills, each with a real outcome.

  3. Part 03 · ~2h

    Late data, backfill, and the schema that changed

    You can rewrite history without double-counting or downtime.

    Skip this if you have run a backfill over a live table and it was fine.

    1. SectionIncremental processing6 minOnly what is new. What "new" means is the whole design.
    2. SectionBackfills: rewriting history safely7 min"Backfill a billion-row table" is a reported question, verbatim.
    3. SectionLate-arriving data & replays6 minNamed in the rejections as the thing designs are missing.
    4. SectionSafe and breaking changes6 minWhich changes are safe is a fact, not a judgement. Know it.
    5. SectionChange a name safely7 min"Schema evolution without downtime" is asked by name. This is how.
    6. ExerciseFourteen months of wrong numbers50 minin the studioFourteen months of wrong numbers. Fix them without a second outage.
    7. ExerciseThe payload changed overnight45 minin the studioThe payload changed overnight and nobody told you.
    8. InterviewWhat every event should carry6 minEvery field exists because something downstream is impossible without it.
    9. QuestionsData Pipeline · Architecture & design15 minDesign prompts, out loud. Requirements before components.

  4. Part 04 · ~4h

    When it breaks, and what it costs

    You can name the failure modes and put a number on the design.

    Nothing to skip — cost and failure are how the level gets decided.

    1. SectionSix ways a job fails7 minName them before you are asked. That is the whole seniority signal.
    2. SectionAlerts people answer6 minAn alert nobody answers is not monitoring. Say what pages, and why.
    3. InterviewFail the record or fail the job?6 minA threshold, declared in advance. Not a judgement call at 3am.
    4. ExerciseIt has to be there by six45 minin the studioThe deadline round. What you drop, and what you never drop.
    5. SectionChange a row in a file6 minDelta and Iceberg are on the posting. This is what gets asked.
    6. InterviewCopy-on-Write vs Merge-on-Read6 minFast writes or fast reads. Say which side you bought, and where.
    7. SectionFind the slow stage first6 minMeasure each stage and find the slow one. A method beats configs.
    8. InterviewWhat compaction is, and when to run it6 minMerge-on-Read is a promise to compact. Say who keeps it.
    9. ExerciseIt works, and it costs too much45 minin the studioIt works and costs too much. The most senior version.
    10. StepThe Data Pipeline round, timed80 minin the studioThe whole round: design, canvas, then cost and failure.
    11. QuestionsData Pipeline · Cost & scale15 minEstimate before you build. Bytes per day, then everything else.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Ace the Data Pipeline roundTracked, 4 parts, ~12hSee the other plans17 more, by what you are aiming at