What You'll Master Here
A shuffle deals rows to partitions by key. If one key is a whale, its partition becomes a whale, and the task on it works alone while everyone else waits.
Data skew is when one (or a few) keys have vastly more rows than the rest. After a shuffle (Chapter 17), all rows for a key land in one partition, so a skewed key produces one giant partition, one giant task, while the others finish quickly. That one straggler task holds up the whole stage.
This is one of the most common production performance problems, and one of the trickiest to diagnose because the job "works", just slowly, with most cores idle waiting on one task. This chapter shows how to spot it in the Spark UI and three fixes: salting, isolate-and-broadcast, and AQE skew-join handling.
With a diagram of a skewed shuffle (one fat partition) and a worked salting example. Skew is a rite-of-passage debugging skill, by the end you will recognise it instantly.
A shuffle deals rows to partitions by key. If one key is a whale, its partition becomes a whale, and the task on it works alone while everyone else waits.
Skew causes the maddening "99% of tasks done in seconds, last task runs for an hour" pattern. It is a top real-world Spark issue and a favourite interview scenario, knowing the fixes is a strong signal of hands-on experience.
- data skew
- A few keys holding far more rows than the rest, after a shuffle.
- straggler task
- One task that runs far longer than its peers, stalling the stage.
- salting
- Adding randomness to a skewed key to split it across many partitions.
- hot key
- A single key (e.g., null, "unknown", a mega-customer) with outsized volume.
Adding more executors to fix a skewed stage. The bottleneck is one task on one core; extra executors sit idle, throwing resources at skew does not help.
Suspect skew when one task in a stage runs far longer than the rest.
Check the Spark UI task duration/shuffle-read distribution (max vs median).
Look for hot keys: nulls, defaults, and a few dominant entities.
Skew is a load-balancing failure: the shuffle put too much on one partition. The fixes either spread the hot key out (salting), handle it separately (isolate-and-broadcast), or let Spark split it (AQE).
Data skew concentrates a hot key into one giant partition and one straggler task, stalling the stage. Diagnose it by uneven task durations; fix it with salting, isolate-and-broadcast, or AQE skew handling, not more executors.
- Why does a skewed key create a single slow task after a shuffle?
- Why does adding executors not fix a skewed stage?