What You'll Master Here
If a table is small, broadcast it (copy to all). If you reuse the same data per task, broadcast it as a variable. If you join the same key repeatedly, bucket it once on disk. Each avoids moving big data.
This chapter is a toolkit of shuffle-avoiding and data-sharing techniques. The headline act is the broadcast join: when one table is small, ship it to every executor so the big table can be joined locally, no shuffle at all. It is the single most effective join optimisation in Spark.
Alongside it, two shared-variable mechanisms, broadcast variables (read-only data sent once to every executor) and accumulators (write-only counters aggregated on the driver), and bucketing, which pre-shuffles data on write so repeated joins on the same key skip the shuffle entirely.
With diagrams of broadcast-vs-shuffle joins and the bucketing layout, plus runnable examples. These are the techniques that separate a job that shuffles needlessly from one that does not.
If a table is small, broadcast it (copy to all). If you reuse the same data per task, broadcast it as a variable. If you join the same key repeatedly, bucket it once on disk. Each avoids moving big data.
Broadcast joins and bucketing remove shuffles, the dominant cost (Chapter 17). Broadcast variables and accumulators are the correct way to share data and collect metrics across executors. All four come up constantly in real tuning and interviews.
- broadcast join
- Joining by copying the small table to every executor, avoiding a shuffle.
- broadcast variable
- A read-only value shipped once to each executor and reused by its tasks.
- accumulator
- A write-only counter tasks add to, aggregated on the driver.
- bucketing
- Pre-partitioning data by a key on write so later joins avoid shuffling.
Broadcasting a table that is too large. Every executor must hold a full copy; an oversized broadcast causes driver/executor OOM, broadcast only small tables.
Broadcast the small side of a large–small join.
Use broadcast variables for lookup data reused across tasks.
Bucket tables joined repeatedly on the same key.
Moving big data is expensive; moving small data (or moving it once) is cheap. Broadcast joins, broadcast variables, and bucketing all replace a big shuffle with a small copy or a one-time layout.
Broadcast joins copy a small table to all executors to avoid a shuffle; broadcast variables share read-only data; accumulators collect metrics; bucketing pre-shuffles on disk. All four trade a big shuffle for something cheap.
- When does a broadcast join avoid a shuffle, and what is the risk?
- What is the difference between a broadcast variable and an accumulator?