What You'll Master Here
A join gathers matching keys together. By default that means shuffling both sides (slow). If one side is small, you can instead copy it to every executor (a broadcast join) and skip the shuffle entirely.
Joins combine two DataFrames on a key, and they are both essential and the most common place Spark jobs go slow. This chapter covers the join types you will use, how Spark physically executes a join (the strategies), the single biggest optimization (the broadcast join), and the pitfalls that turn a join into a runaway job.
The key realisation: a normal join is a wide transformation. It shuffles both sides of the data across the cluster so matching keys land together, exactly the expensive operation from Chapter 5. Most join tuning is about avoiding or shrinking that shuffle.
Everything is concrete: two small tables joined every which way, with inputs and outputs, plus diagrams of the shuffle join and the broadcast join.
A join gathers matching keys together. By default that means shuffling both sides (slow). If one side is small, you can instead copy it to every executor (a broadcast join) and skip the shuffle entirely.
"My Spark job is slow" is, more often than not, a join problem, a huge shuffle, a skewed key, or a missed broadcast. Joins are also a staple of interviews. Getting them right is core competence.
- join key
- The column(s) two DataFrames are matched on.
- join type
- Which rows survive: inner, left/right/full outer, left semi, left anti.
- shuffle join
- The default: both sides are shuffled by key so matches meet (wide, slow).
- broadcast join
- Copy a small table to every executor and join locally, no shuffle.
Treating every join as cheap. A default join shuffles both sides; on large data that shuffle often dominates the whole job.
Filter and select before joining, so less data is shuffled.
Broadcast the small side whenever one table is small.
Check the join key for skew and nulls before blaming Spark.
A join is a shuffle unless you can broadcast. Most join performance work is "shrink the data, then either broadcast the small side or fix the skewed key".
Joins match two DataFrames on a key. By default they shuffle both sides (wide and slow); broadcasting a small side avoids the shuffle. Most slow jobs are slow joins.
- Why is a default join a wide transformation?
- When can you avoid the join shuffle entirely?