D8LooP
Start building

Interviewing

How to Prepare for the Apache Spark Interview Round

Explain what Spark ran, from the plan. Shuffles, joins, skew, AQE and the tuning follow-ups.

4 parts34 steps~8h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can narrate any Spark job from its plan, and fix it with evidence.

Round day Every answer backed by a plan

02

The plan, 4 parts

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Part 01 · ~2h

    Say what Spark actually ran

    You can say which lines ran jobs and where each stage boundary is.

    Skip this if you already narrate jobs and stages from the Spark UI.

    1. InterviewThe three components, named6 minThe opener. Name all three, and what each does.
    2. SectionNarrow vs Wide Transformations6 minEvery stage boundary comes from one wide dependency.
    3. ExerciseWhy did my accumulator double? Counters inside lazy transformations20 minin the studioLazy evaluation, proved by a counter that doubles.
    4. InterviewHow many DAGs does a script build?6 minSeveral actions, several jobs, each one recomputed.
    5. SectionKey Optimizations & Reading explain()7 minRead a physical plan bottom up, out loud.
    6. ExerciseReading explain("formatted") end to end on a three-table query25 minin the studioThe debugging-round format: walk a formatted plan node by node.
    7. InterviewLogical vs physical plan6 minWhat each plan decides. Asked in almost every Spark screen.
    8. QuestionsApache Spark · Conceptual & architecture15 minFundamentals, said cleanly, each claim tied to evidence.

  2. Part 02 · ~2h

    The shuffle, and the join behind it

    You can name the join strategy and count the shuffles in any plan.

    Skip this if you can say why a join shuffled, from its plan.

    1. SectionAnatomy of a shuffle7 minEvery Spark performance answer starts at the Exchange.
    2. Exerciserepartition("country") before groupBy("country"): saved shuffle or wasted one?20 minin the studioCount the Exchanges before believing a comment about them.
    3. SectionHow Spark executes joins7 minFour strategies. Know what each one needs and costs.
    4. ExerciseautoBroadcastJoinThreshold compares an estimate: flip a join with select() and one setting25 minin the studioTwo selected columns flip the strategy. Explain the estimate.
    5. ExerciseAggregate before the join or after it? Let the shuffle sizes decide25 minin the studio"Aggregate before joining": test the rule, find where it flips.
    6. ExerciseTop N per group: when Spark's WindowGroupLimit shortcut switches off25 minin the studioTop-N per group, the coding classic, and what makes it slow.
    7. InterviewTask not serializable6 minA closure that drags the driver along. Reported constantly.
    8. InterviewHow many partitions for 1 TB?6 minArithmetic out loud: bytes over target partition size.
    9. QuestionsApache Spark · Plan reading15 minWhere each Exchange is, and what crosses it.

  3. Part 03 · ~2h

    Skew, and what AQE does about it

    You can prove skew from task metrics and say what AQE changed.

    Skip this if you can explain why AQE left a skewed join alone.

    1. SectionDetecting skew in the Spark UI6 minMax against median task time. Skew, or a slow cluster?
    2. ExerciseWhere does skew show up? Reading task metrics, not the DAG20 minin the studioIndia dominates, yet the groupBy is even. Explain why.
    3. ExerciseOne user owns 30% of the events: which operations does a hot key hurt?25 minin the studioCount even, join skewed, AQE silent. Three facts, one cause.
    4. SectionSalting, isolate-and-broadcast & AQE7 minSalt, isolate, or let AQE split. Know what each costs.
    5. ExerciseSalting a skewed join key by hand: flatter tasks, same answer30 minin the studioSalt it, prove the answer held, then question the need.
    6. SectionWhat AQE fixes at runtime6 minThe three runtime rewrites, and the settings behind each.
    7. ExerciseThe plan Spark starts with vs the plan it actually ran20 minin the studioisFinalPlan=false: find the plan that actually ran.
    8. InterviewWalk me through fixing skew6 minThe story form: symptom, evidence, cheapest fix, the number.
    9. QuestionsApache Spark · Debug & fix15 minTwenty failures diagnosed aloud: symptom, cause, fix.

  4. Part 04 · ~2h

    The tuning follow-ups

    You can defend a partition count, a cache and a cluster size with numbers.

    Skip this if you size clusters and partitions from the data, not habit.

    1. SectionSizing partitions well6 minWhere partition counts come from, and the size to aim for.
    2. ExerciseIf AQE coalesces anyway, does spark.sql.shuffle.partitions still matter?20 minin the studioWhat spark.sql.shuffle.partitions still controls once AQE is on.
    3. ExerciseWhat a spill looks like, and how to make it go away25 minin the studioSpill with no error. Remove it without adding memory.
    4. ExerciseThree reports, one expensive input: where exactly should cache() go?25 minin the studioWhere to cache, how wide, and proof it paid.
    5. InterviewWhy small files hurt, step by step6 minSay what Spark does per file, not just "bad".
    6. InterviewExecution vs storage memory6 minThe unified pool, and what gets evicted first.
    7. InterviewSize a cluster for 1 TB6 minSubtract the overhead first, then divide. Out loud.
    8. QuestionsApache Spark · Performance tuning15 minEvery fix with its cost, and how you measured it.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Ace the Spark roundTracked, 4 parts, ~8hSee the other plans16 more, by what you are aiming at