Data Pipeline Interview Questions for Data Engineers

01

The 5 kinds of Data Pipeline question

Cut by what a question asks you to DO, not by the topic it names. The same topic scores differently when you have to explain it, build it, or work out why it broke — and those three fail separately.

• 25ConceptualThe distinctions that decide an architecture. What idempotency buys you, why exactly-once is a property of the write rather than of the transport, and what a watermark is actually tracking.Batch vs streaming · ETL vs ELT · Idempotency & exactly-once · Incremental loads & watermarks→
• 25Architecture DesignA one-line prompt — "ingest clickstream and serve it to analysts within an hour" — and a whiteboard. Graded on the boundaries you draw and the assumptions you state, not on the tools you name.CDC & ingestion patterns · Orchestration & dependencies · Storage formats · SLAs & observability→
• 24Failure & Recovery ScenariosThe job died mid-write. The upstream re-sent yesterday. A schema changed at 3am. For each one: what state is the system in now, what happens on rerun, and what you would have built so the answer was boring.Idempotency & exactly-once · Backfills & reprocessing · Late / out-of-order data · Schema evolution→
• 23Trade-offs & Tool ChoiceKafka or a nightly extract, Airflow or the warehouse scheduler, streaming or a five-minute batch. The signal is in what you give up and how you would know later that you chose wrong.Batch vs streaming · ETL vs ELT · Orchestration & dependencies · Data contracts & quality gates→
• 22Cost & Scale EstimationNumbers on the back of an envelope: rows per day, bytes per row, what that costs to store, scan and reprocess, and which part of the design falls over first at ten times the volume.File layout & compaction · Storage formats · Multi-region / DR · PII & security→

02

The ones that keep coming up

Answered here, in short. The full reasoning, the wrong answer and the follow-up are one click away.

1. 01Does a streaming platform like Kafka or Kinesis guarantee that events arrive in order?Ordering is guaranteed within a partition, not across the stream. Since records with the same key go to the same partition, you get ordering per key — and only for the key you chose.→
2. 02How do you choose between INSERT INTO, MERGE and INSERT OVERWRITE when loading into a target table?Append for immutable facts, merge for entities that can change or arrive again, overwrite for recomputing a bounded slice. The deciding question is what happens when the job runs twice.→
3. 03A Flink job reports 90% backpressure and consumer lag on its Kafka topic is growing. What is backpressure, and what would you do about it?Backpressure is a slow stage signalling upstream to send less, so the slowness propagates as a rate limit rather than as memory exhaustion. In a bus-based pipeline you usually do not have it end to end — the bus absorbs instead, and lag is what you watch.→
4. 04What is a session window with a 30-minute gap, and how does it differ from a tumbling window?A session window has no fixed length: it ends after a gap of inactivity for that key. So its duration is data-dependent, its state is held per key for as long as the key stays active, and unlike a tumbling window it has no guaranteed end.→
5. 05Which medallion layer standardises date formats and handles nulls, and what does the raw Parquet in bronze keep untouched?Raw stays as close to the source as possible, unchanged. Standardisation and null handling belong in the curated layer, written once and shared, so that a mistake there can be fixed by reprocessing rather than by re-fetching.→
6. 06Where does the dead-letter queue sit in a Kafka or Kinesis pipeline, and what does a dead-lettered record need to carry?There is more than one. Downstream consumers need a DLQ for records they cannot process, and the producer needs its own failure store for records that fail validation or exhaust retries before ever reaching the stream.→
7. 07Retries mean the same event_id can arrive twice, and out-of-order delivery means version 3 can arrive after version 4. How do you make the consumer safe against both?The event id handles duplicates; a producer-assigned version or sequence number handles ordering. Apply an update only when its version is higher than the one already stored, so a late-arriving old event is rejected rather than applied.→
8. 083,000 records in a batch of a million fail validation. Do you fail the whole job, or process the good ones and quarantine the rest?Separate record-level errors from system-level ones. A few bad records get quarantined while the good ones flow; a failure rate above a declared threshold fails the job, because at that point the input itself is suspect.→
9. 09A leadership review starts in 60 minutes and the report feeding it is showing wrong numbers. What do you do, in what order?Establish what is wrong and how badly it affects the meeting, tell the stakeholders immediately, and put up the last validated snapshot as a fallback while the real fix runs in parallel. Root cause comes after the meeting.→
10. 10Explain Copy-on-Write and Merge-on-Read in a table format like Delta, Iceberg or Hudi, and say which you would put in each medallion layer.Copy-on-Write rewrites whole files on update — slow writes, clean reads. Merge-on-Read appends delta logs and merges at query time — fast writes, slower reads. Merge-on-Read in bronze where writes dominate, Copy-on-Write in silver and gold where reads do.→
11. 11Kinesis offers on-demand and provisioned capacity modes. Which do you start with, and what would make you switch after 3 months of traffic data?Start on-demand while the traffic shape is unknown — it removes shard management entirely. Move to provisioned once the pattern is predictable and the throughput is high enough that the premium stops being worth the convenience.→
12. 12You have 4 event families to publish — orders, payments, audit records and clickstream. One Kafka topic or several, and why?Split by business criticality and traffic shape, not by convenience. Clickstream and transactional events do not belong on the same stream, because a clickstream burst should not be able to throttle payments.→
13. 13What is compaction, why does a Merge-on-Read table in Hudi or Iceberg need it, and how do you decide when to run it?Compaction merges base files and their accumulated delta logs into new clean files, so readers stop paying a merge cost. Minor compaction folds the logs together, major compaction rewrites the base — and it runs on a trigger, never continuously.→
14. 14You need to ingest 200,000 events per second at about 2 KB each. How do you size the stream, and what would you check besides the total?Total throughput is 400 MB/s, which sets a floor on partition count. But partitions are assigned by key hash, so you also have to check the distribution of keys — one hot key saturates one partition regardless of headroom elsewhere.→
15. 15You are putting events on Kafka or Kinesis. How do you choose the partition key, and what makes a key a bad one?Choose it from the ordering you need: the entity whose events must stay in sequence. Bad keys are the ones with too few distinct values — a constant, a status, a country — because they collapse the stream onto a few partitions.→
16. 16You are defining the event contract for a new Kafka topic. Beyond the payload, what metadata does every event carry — event_id, event_time, what else — and what does each field buy you later?An envelope with event_id, event_type, event_time, source, schema_version and a trace id, wrapping an opaque payload. Each one exists to make a specific downstream job possible — dedup, ordering, replay, lineage or debugging.→
17. 17Should schema validation happen in the producer before an event is published to Kafka, or downstream in the Spark job that reads it?Cheap contract checks belong at the producer, before anything is published; expensive semantic checks belong downstream. The stream should not become a dump pipe, because everything that enters it reaches every consumer.→
18. 18Parquet files are immutable. So what actually happens when you run an UPDATE against a Delta or Iceberg table, and how does the engine find the right file among 10 million rows?Under Copy-on-Write the engine rewrites the whole data file containing the matching rows and atomically swaps it into the table metadata. It finds the file from per-file min/max statistics and the partition path, not by scanning.→
19. 19A nightly batch job feeds a critical report and the business now wants it within 5 minutes. How do you get from batch to streaming without a big-bang cutover?Run both in parallel, compare their outputs on the same inputs until they agree, move consumers over gradually, and keep the batch path as the reconciliation baseline rather than deleting it.→

03

What Data Pipeline interviews actually cover

• 01Batch vs streaming
• 02ETL vs ELT
• 03CDC & ingestion patterns
• 04Idempotency & exactly-once
• 05Incremental loads & watermarks
• 06Late / out-of-order data
• 07Backfills & reprocessing
• 08Schema evolution
• 09Data contracts & quality gates
• 10Orchestration & dependencies
• 11File layout & compaction
• 12Storage formats
• 13SLAs & observability
• 14Lineage
• 15PII & security
• 16Multi-region / DR

04

Learn it, then practise it