Spark Performance Tuning: Make Slow Spark Jobs Fast

01

When you finish, you can

You can make a slow Spark job cheaper and prove the answer did not change.

Tuned Faster, and provably the same answer

02

The plan, 4 levers

Ordered by what each one buys you, not by topic. Every step says why it is here.

1. Lever 01 · ~2h

   Read the evidence before the code

   You can find the metric that explains a slow job before changing anything.

   Skip this if you open the Stages tab before the notebook.

   1. SectionReading the Spark UI7 minWhich tab answers which question, and what to read first.→
   2. ExerciseThree ways to compute customer spend: rank them by shuffle size before anyone guesses20 minin the studioRank three pipelines by shuffle write before reading their code.
   3. ExerciseThe plan says PushedFilters. Did it skip anything? Read the numbers25 minin the studioPer-node row counts prove what pushdown really skipped.
   4. SectionBottleneck signatures, configs & small files7 minEach bottleneck leaves a signature. Learn the short list.→
   5. ExerciseFind the four suspects in a slow pipeline (one of them is innocent)30 minin the studioFour suspects, one verdict each, every verdict backed by a metric.
   6. InterviewWhere to look in the Spark UI6 minThe opener of every debugging round. Say your first page.→
   7. InterviewClassify the failure first6 minName the failure class before chasing its cause.→
   8. QuestionsApache Spark · Conceptual & architecture15 minJobs, stages, tasks, executors: the vocabulary evidence is read in.→

2. Lever 02 · ~2h

   Move fewer bytes

   You can shrink or remove a shuffle without changing the answer.

   Skip this if you cut shuffle size before adding executors.

   1. SectionReducing and avoiding shuffles6 minShuffle less, shuffle once, or do not shuffle at all.→
   2. ExerciseAggregate before the join or after it? Let the shuffle sizes decide25 minin the studioAggregate before the join, and find when it stops helping.
   3. ExerciseThree window columns, two or three shuffles? The order of withColumn matters20 minin the studioReorder window columns to share one shuffle. Same results.
   4. ExerciseautoBroadcastJoinThreshold compares an estimate: flip a join with select() and one setting25 minin the studioA size estimate decides the join. Move it deliberately.
   5. ExerciseFour dashboard metrics: four queries, one union, or one pass?20 minin the studioFour scans become one. Say what the single pass gives up.
   6. ExerciseA three-table enrichment with zero shuffles, and what brings them back20 minin the studioThree tables, zero shuffles, until the dimension is an aggregate.
   7. InterviewRemoving two shuffles with a hint6 minA before-and-after pair: fewer Exchanges, identical rows.→
   8. InterviewA tuning change that changed the number6 minThe tuning mistake that matters most: faster, and wrong.→
   9. QuestionsApache Spark · Plan reading15 minFor any plan: where it shuffles, and how to shuffle less.→

3. Lever 03 · ~3h

   Fix the one slow task

   You can prove skew, let AQE fix what it can, and salt the rest.

   Skip this if you know why AQE ignored your skewed join.

   1. SectionDetecting skew in the Spark UI6 minProve skew before you fix it. Two metrics decide.→
   2. ExerciseOne big country, one slow task: skew inside a window25 minin the studioRows per post-shuffle partition, and what coalescing hides.
   3. ExerciseOne user owns 30% of the events: which operations does a hot key hurt?25 minin the studioWhy the count is even and the join is not.
   4. SectionWhat AQE fixes at runtime6 minWhat AQE splits, merges and converts at runtime.→
   5. ExerciseMake AQE split a skewed join partition, and learn why it usually does not30 minin the studioLower the thresholds until AQE splits the hot partition.
   6. SectionSalting, isolate-and-broadcast & AQE7 minSalting and isolation, and what each one costs.→
   7. ExerciseSalting a skewed join key by hand: flatter tasks, same answer30 minin the studioSalt the join, prove identical results, then question it.
   8. InterviewThe order to try skew fixes6 minCheapest first: broadcast, filter the key, AQE, then salt.→
   9. QuestionsApache Spark · Debug & fix15 minSymptom, metric, cause and fix, twenty times, out loud.→

4. Lever 04 · ~2h

   Spend memory on purpose

   You can size partitions, caches and clusters from the data.

   Skip this if your partition counts and cluster sizes come from arithmetic.

   1. SectionThe executor memory model7 minOne pool for execution and storage. Every cache borrows from it.→
   2. ExerciseWhat a spill looks like, and how to make it go away25 minin the studioRemove a spill by changing partition count, not memory.
   3. ExerciseIf AQE coalesces anyway, does spark.sql.shuffle.partitions still matter?20 minin the studioWhat the partition setting still decides with AQE on.
   4. ExerciseOne aggregate, two joins: when cache pays off and when Spark reuses work on its own25 minin the studioCache, or fold two reports into one query instead.
   5. ExerciseThree reports, one expensive input: where exactly should cache() go?25 minin the studioFull-width against narrow cache, measured by cached size.
   6. Exercisecollect() on something big: where the driver says no20 minin the studioThe driver has limits too. Hit one safely.
   7. InterviewHow many partitions for 1 TB?6 minBytes over target size, then check it against the cores.→
   8. InterviewSize a cluster for 1 TB6 minSubtract the overhead out loud before you divide.→
   9. QuestionsApache Spark · Performance tuning15 minEvery fix with its cost, and the measurement behind it.→

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.