D8LooP
Start building

Interviewing

How to Prepare for the Data Pipeline Design Round

The diagram is the easy half. This is the half that gets people rejected.

4 parts37 steps~9h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can design a pipeline and defend the reruns, the late data and the cost.

Design day You can defend a design under questioning

02

The plan, 4 parts

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Part 01 · ~2h

    Requirements before boxes

    You ask what the pipeline is for before choosing anything to build it.

    Skip this if you ask about latency, volume and consumers unprompted.

    1. SectionThe requirements-first method7 minNaming tools before requirements is the documented top elimination.
    2. SectionThe questions to ask first6 minLatency, volume, consumers, retention. Ask before you draw.
    3. SectionThe constraints nobody writes down6 minBudget, team size, what already exists. These decide more than tech.
    4. SectionLatency: a budget, not a dial6 minTurn "real time" into a number before you agree to build it.
    5. SectionChoosing without overbuilding6 minChoosing streaming when batch would do is a real mark against you.
    6. InterviewChoosing a partition key6 minOne choice sets ordering, distribution, hotspots and scale. Say all four.
    7. ExerciseRealtime events pipeline45 minin the studioA prompt lifted from real loops: batch or streaming, and why.
    8. InterviewWhat ordering a stream gives you6 minPer partition, never global. A flat "yes" designs a reordering bug.
    9. QuestionsData Pipeline · Conceptual15 minThe vocabulary the rest of the round is conducted in.

  2. Part 02 · ~2h

    Make it safe to run twice

    You can say what happens on a rerun without checking the code.

    Skip this if every load you design is a merge or a partition overwrite.

    1. SectionAt-most, at-least, exactly-once6 minThree guarantees, three costs. Pick one and say what it buys.
    2. SectionIdempotency: safe to repeat7 minThe word rejection notes use when it is missing. Say it early.
    3. SectionDeduplication & idempotency keys6 minWho assigns the key decides whether dedup works at all.
    4. SectionAppend, upsert, overwrite6 minThree write modes, and only one of them duplicates on a rerun.
    5. InterviewAppend, merge or overwrite6 minRun it twice. Append duplicates; the other two converge.
    6. ExerciseThe fact arrived first45 minin the studioThe fact beat its dimension. What does your load do about it?
    7. ExerciseYou fixed it. Who already used it?50 minin the studioYou fixed it and replayed. Who already used the wrong numbers?
    8. InterviewDeduplicating with a version6 minDedup handles copies. Only a version handles arriving out of order.
    9. QuestionsData Pipeline · Failure & recovery15 minTwenty reruns, replays and backfills, each with a real outcome.

  3. Part 03 · ~2h

    Late data, backfill, and the schema that changed

    You can rewrite history without double-counting or downtime.

    Skip this if you have run a backfill over a live table and it was fine.

    1. SectionIncremental processing6 minOnly what is new. What "new" means is the whole design.
    2. SectionBackfills: rewriting history safely7 min"Backfill a billion-row table" is a reported question, verbatim.
    3. SectionLate-arriving data & replays6 minNamed in the rejections as the thing designs are missing.
    4. SectionBackward, forward & breaking changes6 minWhich changes are safe is a fact, not a judgement. Know it.
    5. SectionShipping a breaking change anyway7 min"Schema evolution without downtime" is asked by name. This is how.
    6. ExerciseFourteen months of wrong numbers50 minin the studioFourteen months of wrong numbers. Fix them without a second outage.
    7. ExerciseThe payload changed overnight45 minin the studioThe payload changed overnight and nobody told you.
    8. InterviewWhat every event should carry6 minEvery field exists because something downstream is impossible without it.
    9. QuestionsData Pipeline · Architecture & design15 minDesign prompts, out loud. Requirements before components.

  4. Part 04 · ~2h

    When it breaks, and what it costs

    You can name the failure modes and put a number on the design.

    Nothing to skip — cost and failure are how the level gets decided.

    1. SectionThe failure modes you must expect7 minName them before you are asked. That is the whole seniority signal.
    2. SectionSLIs, SLOs & alerts people answer6 minAn alert nobody answers is not monitoring. Say what pages, and why.
    3. InterviewFail the record or fail the job?6 minA threshold, declared in advance. Not a judgement call at 3am.
    4. ExerciseIt has to be there by six45 minin the studioThe deadline round. What you drop, and what you never drop.
    5. SectionCopy-on-Write vs Merge-on-Read6 minDelta and Iceberg are on the posting. This is what gets asked.
    6. InterviewCopy-on-Write vs Merge-on-Read6 minFast writes or fast reads. Say which side you bought, and where.
    7. SectionThe job is slow: where to look6 minSkew, shuffle, small files. A method beats a list of configs.
    8. InterviewWhat compaction is, and when to run it6 minMerge-on-Read is a promise to compact. Say who keeps it.
    9. ExerciseIt works, and it costs too much45 minin the studioIt works and costs too much. The most senior version.
    10. QuestionsData Pipeline · Cost & scale15 minEstimate before you build. Bytes per day, then everything else.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Ace the Data Pipeline roundTracked, 4 parts, ~9hSee the other plans16 more, by what you are aiming at