What You'll Master Here
Partitions are the slices; cores are the knives. You want enough slices to keep every knife busy, each slice a comfortable size, and, on disk, slices arranged so you can grab only the ones you need.
A partition is a chunk of your data, and it is the unit of parallelism: one partition becomes one task on one core (Chapter 15). So how your data is partitioned directly sets how parallel, and how fast, your job runs. This chapter is about controlling that deliberately.
You will learn the two tools to change partition count, repartition (a full shuffle, can increase or decrease) and coalesce (no shuffle, only decreases), how to size partitions so tasks are neither tiny nor huge, and partition pruning, where a smart layout on disk lets Spark skip reading whole directories.
With diagrams contrasting repartition vs coalesce and a worked partitioned-write that enables pruning. This is the lever you reach for most often in tuning.
Partitions are the slices; cores are the knives. You want enough slices to keep every knife busy, each slice a comfortable size, and, on disk, slices arranged so you can grab only the ones you need.
Partition count is the single most impactful knob for parallelism and memory. Too few starves the cluster; too many drowns it in overhead; the wrong on-disk layout reads data you never needed.
- partition
- A chunk of the dataset processed by a single task.
- repartition
- Changes partition count via a full shuffle; can increase or decrease.
- coalesce
- Reduces partition count without a full shuffle by merging existing ones.
- partition pruning
- Skipping whole on-disk partitions that a filter rules out.
Ignoring partition count and accepting whatever the source gives you. You inherit too-few or too-many partitions, leaving cores idle or overwhelmed with overhead.
Aim for partitions a small multiple of total cores, sized ~100–200 MB each.
Use coalesce to shrink, repartition to grow or rebalance.
Partition on-disk data by a column you filter on, to enable pruning.
Partitions are the atoms of Spark parallelism. Control how many you have (repartition/coalesce) and how they sit on disk (partitionBy), and you control speed and how much data you even read.
Partitions are the unit of parallelism. Sizing them well (repartition/coalesce) and laying them out on disk for pruning (partitionBy) are among the highest-leverage tuning moves in Spark.
- Why does partition count directly determine parallelism?
- What two things can go wrong with too few, or too many, partitions?