Partitioning & Parallelism

In the chapter "Spark Architecture: Driver, Executors & Cluster Managers", one partition became one task. So the number of partitions decides how much work can run at the same time. This chapter shows how to see that number and change it.

The word partition has two meanings in Spark work. Keep them apart.

The chapter "Storage Layers & File Formats" taught the second meaning: date folders and small files. This chapter teaches the first meaning, then joins the two.

This chapter has four parts.

1. How partitions and cores work together, and how to see your partitions.
2. Where the partition count comes from, and what count to aim for.
3. How repartition and coalesce change the count.
4. How partitions decide the files and folders you write.

We use the shop again: 6 orders, and a cluster with 2 executors and 2 cores each. That gives 4 slots. A slot runs one task at a time.

Spark cuts a table into pieces. Each piece is a partition. Spark runs one task for each partition.

In our model, orders has 3 partitions. Look at which order is in which partition.

That means 3 tasks. The cluster has 4 slots.

A wave is one round of tasks that run at the same time. Here is the single wave.

Slot 4 has nothing to do. This is the first problem: too few partitions leave cores idle.

With 1 partition, 3 of the 4 slots would be idle. More cores would not help.

Now give the same data 8 partitions. There are 8 tasks and 4 slots, so Spark needs two waves.

Every slot is busy now. In a real run, a free slot takes the next task at once. So the waves are a simple picture.

But 8 partitions for 6 orders is too many. At most 6 partitions can hold a row, so at least 2 tasks have nothing to read. Every task also costs time to send to an executor and to start.

You can ask Spark how many partitions a DataFrame has. Use getNumPartitions on its rdd.

print(orders.rdd.getNumPartitions())

To see which row is in which partition, add the function spark_partition_id(). It returns the partition number for each row. Numbers start at 0.

from pyspark.sql import functions as F

orders.select("order_id", "customer_id",
              F.spark_partition_id().alias("pid")).show()

To find uneven partitions, count the rows in each one.

orders.groupBy(F.spark_partition_id().alias("pid")).count().show()

The partition count is not random. It comes from two places: reading files, and shuffles. We start with reading files.

When Spark reads files, it cuts big files into pieces. Each piece is at most spark.sql.files.maxPartitionBytes. The default is 128 MB.

Spark also packs many small files into one partition. The table shows three cases.

Spark also tries to give every core work. So on a cluster with many cores, a small file can give more than 1 partition.

Spark can cut formats like Parquet. It cannot cut a gzip text file, such as orders.csv.gz. So that file becomes one partition, however big it is.

The second place is a shuffle. The chapter "The Shuffle: Spark's Performance Killer" showed the setting spark.sql.shuffle.partitions. Its default is 200.

You can change it for your session.

spark.conf.set("spark.sql.shuffle.partitions", 8)

orders.groupBy("customer_id").sum("amount")

How many partitions should you aim for? A common starting point: first aim for about 100 to 200 MB in each partition. Then check that you have at least 2 to 4 partitions for each core.

These are starting points, not laws. A partition that is too big can fill memory, and then Spark spills it. Spilling means Spark writes data to disk because memory is full.

Check the Spark UI after a run. Look at the task times and the spill numbers. Then change the count and try again.

Spark has two tools to change the partition count. We start from our 3 partitions: 101, 102 | 103, 104 | 105, 106.

coalesce(n) merges existing partitions. It does not shuffle, so no row moves across the network.

It can only lower the count. Here is coalesce(2).

orders.coalesce(2) \
      .select("order_id", F.spark_partition_id().alias("pid")).show()

repartition(n) is different. It does a full shuffle and sends every row to a new partition.

It aims for equal sizes. It can raise or lower the count.

orders.repartition(2) \
      .select("order_id", F.spark_partition_id().alias("pid")).show()

You can also give a column: repartition(n, "customer_id"). Spark sends rows with the same customer_id to the same partition. It does this with a hash of the column.

orders.repartition(2, "customer_id") \
      .select("order_id", "customer_id",
              F.spark_partition_id().alias("pid")).show()

This table puts the three calls side by side.

If you write repartition("customer_id") without a number, Spark uses spark.sql.shuffle.partitions. That default is 200.

A write makes one file for each partition. Each task writes its own file. So the partition count sets the file count.

orders.write.parquet("out/orders")

Too many partitions give too many small files. After a shuffle with the default 200 partitions, a small write can leave up to 200 tiny files. The chapter "Storage Layers & File Formats" showed why small files slow every reader.

The fix is to lower the count just before the write. coalesce is cheap, because it does not shuffle. Now the smallest case: one file.

orders.coalesce(1).write.parquet("out/orders_one")

Now the second meaning of partition: folders on disk. The call write.partitionBy("order_date") writes one folder for each date.

orders.write.partitionBy("order_date").parquet("lake/orders")

A reader can now skip folders. A filter on order_date opens only the matching folder. You met this in the chapter "Storage Layers & File Formats": it is called partition pruning.

spark.read.parquet("lake/orders") \
     .where("order_date = '2026-01-10'").show()

Both meanings meet at the write. Each task writes its own file into every folder it has rows for. So the Spark partitions decide how many files each folder gets.

Imagine a real shop. Each day has orders in all 3 Spark partitions. Then every date folder gets 3 small files.

So repartition by the same column first. Then each date lands in one partition, and each folder gets one file.

One very busy date then becomes one big task and one big file.

orders.repartition("order_date") \
      .write.partitionBy("order_date").parquet("lake/orders")

Four points to keep.

• One Spark partition is one task. Compare the partition count with the number of slots.
• The count comes from file sizes when you read, and from spark.sql.shuffle.partitions after a shuffle.
• coalesce merges without a shuffle and can be uneven. repartition shuffles and can make even sizes.
• One file is written for each partition, in each folder. Repartition by the folder column before partitionBy.

Answer these without looking back.

1. You have 3 partitions and 4 slots. How many slots are idle, and what would you change?
2. A 1 GB file is read with the default 128 MB. How many partitions do you get?
3. Why can coalesce(2) give sizes 2 and 4, while repartition(2) gives 3 and 3?
4. You must write one file from a big job. Why is coalesce(1) risky?