What You'll Master Here
reproduce, observe (Spark UI), localise (which stage), diagnose (which cause), fix the cause, verify. Never change configs blindly.
This chapter ties the performance section together into a repeatable method: how to actually find and fix a slow Spark job. The tool is the Spark UI, and the skill is reading it to locate the bottleneck instead of guessing. By now you know the causes (shuffle, skew, spill, small files, bad partitioning), here you learn the diagnostic loop that points to which one is biting.
We cover a tuning method (measure → find the slow stage → identify the cause → apply the matching fix → re-measure), the handful of configs that actually matter, the bottleneck signatures in the UI, and the pervasive small-files problem with its fixes.
With a diagram of the tuning loop and a worked diagnosis. The meta-lesson: tune from evidence, not superstition, the UI tells you exactly where the time goes.
Tuning is debugging: reproduce, observe (Spark UI), localise (which stage), diagnose (which cause), fix the cause, verify. Never change configs blindly.
Most engineers tune by trial and error. A systematic, UI-driven method finds the real bottleneck fast and is exactly what "walk me through tuning a slow job" interviews want to hear.
- Spark UI
- The web UI showing jobs, stages, tasks, SQL, and storage for diagnosis.
- bottleneck stage
- The stage consuming most of the job's wall-clock time.
- tuning loop
- Measure → localise → diagnose → fix → re-measure, repeated.
- small-files problem
- Too many tiny files/partitions, hurting reads and writes.
Changing several configs at once hoping something helps. You cannot tell what worked (or hurt), and you may mask the real cause, change one thing, measure, repeat.
Always open the Spark UI before changing anything.
Find the slowest stage first; optimise where the time actually is.
Change one variable at a time and re-measure.
Performance tuning is evidence-driven debugging. The Spark UI shows where time goes; your job is to localise the slow stage, read its signature (shuffle? skew? spill?), and apply the matching fix, then verify.
Tuning is a loop: measure with the Spark UI, find the slow stage, read its bottleneck signature, apply the matching fix, and re-measure. Tune from evidence, one change at a time, not from guesses.
- What are the steps of the tuning loop?
- Why change only one config at a time?