Designing batch and streaming data pipelines end to end: ingestion, storage, processing, serving, and the failure paths in between.
What this round really tests
Whether you can take a vague requirement, state assumptions and scale, and reason about trade-offs: batch vs streaming, storage choice, partitioning, idempotency, exactly-once, late/duplicate data, cost, and observability. Senior leveling is usually decided here.
How to prepare
Use one repeatable frame: requirements → ingestion → storage → processing → serving → monitoring.
Always cover late-arriving and duplicate data, idempotent loads, and SLAs.
Know batch vs streaming trade-offs and when each is justified.
Practice one domain pipeline end to end (clickstream, payments, or telemetry).
Running the senior design round
At four to eight years this round usually decides your level, and a large part of the score is how you lead the conversation. Use the same order every time, say each step out loud, and keep to 45 minutes.
Run it in this order
Clarify. Ask about volume, speed, how fresh the data must be, who uses it, and how correct it must be. Write the answers on the board.
Sources. Name each source: a database, files, an API or events. Say how each one changes and how you will read it.
Ingest. Say how data comes in, and what happens to duplicates, order, a replay and a schema change.
Storage. Choose the format, the table format, the partitions, how long you keep data, and how small files are compacted.
Transform. Make every step safe to run twice, and say how you backfill a past day.
Serve. Match the serving layer to the queries and the freshness promise: a warehouse, a small table, or a fast store.
Orchestrate. Say what triggers each step, what waits for what, and how retries work.
Monitor. Check freshness, volume, schema and values. Say who gets the alert, and show the lineage.
Security. Say where personal data is, who may read it, and how it is masked or deleted.
Cost. Give the cost now, in three years and in ten, and the first thing you would cut.
Failure. Name at least three failures for each big part, how you would notice each one, and the fix.
Answer these before they are asked
Senior interviewers come back to the same questions. Each link opens a card with the short answer, the answer that fails, and the next question.
A rubric for practice, not any company's scoring sheet. After each timed design, give every line 0, 1 or 2. Scoring 20 or more out of 24 on all six designs below is a good sign you are ready.
You asked about scale, freshness, users, correctness and budget before drawing anything.
You chose batch or streaming, and tied the choice to a requirement.
Ingestion handles replay, order, duplicates and schema changes.
Storage names the format, table format, partitions, retention and compaction.
Every transform step is safe to run twice and can be backfilled.
The serving layer fits the queries and the freshness promise.
Monitoring covers freshness, volume, schema and values, with alerts and lineage.
You said where personal data is and who can read it.
You gave the cost over time and the first thing you would cut.
You named at least three failures for each big part, with the fix.
You explained what you would buy and what you would build, and why.
You led the conversation, handled interruptions and kept to time.
Six designs to practise
Do each one twice: once slowly to learn the shape, then once in 45 minutes, out loud.
Clickstream at scaleIngest about 10 billion app events a day, with peaks near 200,000 a second, and give analysts sessions and daily numbers.
What it tests. Partition keys and retention in Kafka, sessions and duplicates by event time, a table layout by event date, and a schema change from the app team.
A database into the warehouse with CDCCopy a busy Postgres or MySQL database into the warehouse every few minutes, including deletes, with a snapshot to start.
What it tests. Snapshot plus stream without a gap, order by log position, hard deletes, merges on a key, and replication lag.
