What You'll Master Here
Parallelism makes big data tractable; skew and shuffle are what make parallel jobs slow. A job is only as fast as its slowest task, and moving data is the dominant cost.
A pipeline that works on a gigabyte can collapse on a terabyte. Scaling means spreading work across many machines; performance means doing it without wasting time or money. Almost every "why is this job slow?" comes down to a handful of ideas, and this chapter is those ideas with the numbers attached.
A supermarket checkout is the everyday version. Open twenty tills and twenty customers are served at once — that is parallelism. But one customer with three trolleys holds up their till while the other nineteen stand idle, and the queue is only finished when they are: that is skew. And if the shop reorganises so all the frozen goods must be carried to one corner before anyone can count them, that walking is the most expensive part of the day: that is shuffle.
By the end you will know what a cluster is actually made of, how to size partitions with a number rather than a hunch, what a shuffle physically does, how to fix skew with real code, how to read a slow job off the Spark UI in five minutes, and what each fix is worth in pounds.
Parallelism makes big data tractable; skew and shuffle are what make parallel jobs slow. A job is only as fast as its slowest task, and moving data is the dominant cost.
Performance problems at scale are rarely about the code being wrong. They are about skew, shuffle and file layout. Diagnosing those turns an unexplained four-hour job into a thirty-five-minute one, which is one of the highest-leverage skills in the field.
- cluster
- One driver plus a set of executors. The driver plans and hands out work; the executors do it. Marlow runs 20 executors with 4 cores each — 80 places work can happen at once.
- worker / executor
- A process on one machine with a fixed number of cores and a fixed amount of memory. "Worker" is the loose word; "executor" is what Spark calls it.
- parallelism
- Processing many partitions at the same time across those cores.
- partition skew
- Work spread unevenly, so one task does far more than the rest and everyone waits for it.
- shuffle
- Moving rows across the network so that all rows with the same key end up together — needed by joins, group-bys and distinct.
- small-files problem
- So many tiny files that the per-file overhead costs more than reading the data.
- Spark UI
- The web interface every Spark job exposes, with a page per job, stage and task. Every diagnosis in this chapter is read from it.
Assuming "add more machines" always makes a job faster. With skew, extra workers sit idle waiting for one task. With too many tiny partitions, the driver is the bottleneck and executors starve. Hardware fixes neither.
Tuning before looking. Three numbers on the Spark UI stage page separate every cause in this chapter. Guessing instead costs a day and usually changes the wrong knob.
Read max task duration against median before changing anything.
Balance the work and shrink the shuffle before sizing the hardware.
Keep file and partition sizes near 128 MB in both directions.
Parallelism is easy; balancing it is hard. A distributed job is only as fast as its slowest task, and data movement is the dominant cost. Tune those two and most performance problems disappear.
The next topic gives you five words — cluster, partition, task, stage, job — that the rest of the chapter leans on constantly. Ten minutes there makes everything after it concrete instead of abstract.
Scaling means parallelising across machines; performance means defeating skew, shuffle and small files — the handful of causes behind almost every slow distributed job.
- Explain why a job that was fine on a gigabyte falls over on a terabyte.
- Name the two villains that make parallel jobs slow, using the checkout analogy.