What You'll Master Here
how big is the data, and how big is the cluster. From those you derive partitions, cores, memory, and executor count, then verify in the Spark UI.
Spark has hundreds of configuration properties, but only a handful decide whether your job is fast, slow, or dead on arrival. This chapter is the one you will come back to: the configs that actually matter, what each one does and why, and, crucially, the formulas to size them for your data instead of copying numbers off the internet.
We tie together the execution and memory internals from the whole performance section (partitions, cores, the shuffle, executor memory, AQE) into concrete, defensible settings. The goal is not to memorise magic numbers but to derive them: given your data size and cluster, you should be able to compute a sensible starting configuration and explain every value.
You get the precedence rules (where a config can be set and which wins), the resource-sizing formula, worked end-to-end configurations for 10 GB, 500 GB, and 10 TB jobs as copy-paste JSON, and a method to adapt them to any size. This is the chapter that turns "it works on my laptop" into "it runs on the cluster, sized on purpose."
Configuration is arithmetic, not guesswork. Two inputs drive everything: how big is the data, and how big is the cluster. From those you derive partitions, cores, memory, and executor count, then verify in the Spark UI.
Most production Spark pain, OOM, idle cores, endless shuffles, runaway cost, is a sizing problem, not a code problem. Being able to calculate a configuration from data size and cluster shape is one of the most valuable, and most tested, Spark skills.
- configuration property
- A spark.* setting controlling resources, parallelism, memory, or behaviour.
- resource configs
- executor/driver cores and memory, executor count, memory overhead.
- parallelism configs
- Partition counts: spark.sql.shuffle.partitions, default.parallelism.
- precedence
- The order in which config sources override one another.
Copying a configuration block from a blog without recomputing it for your data and cluster. Numbers tuned for someone else's 2 TB on 50 nodes will under- or over-provision your job, derive your own.
Start every config from two numbers: data size and cluster size.
Set only the configs that matter; leave the rest at defaults (let AQE help).
Verify the chosen config against the Spark UI and adjust.
You do not guess Spark configs, you calculate them. Data size sets partitions; cluster shape sets cores, memory, and executors. Everything else is a small set of behaviour flags. Compute, then verify in the UI.
A good Spark configuration is derived, not copied: from your data size and cluster shape you compute partitions, cores, memory, and executor count, set a few behaviour flags (AQE, broadcast threshold), and verify in the UI. This chapter gives the formulas and worked examples.
- What two inputs drive almost every Spark configuration decision?
- Why is copying a config block from the internet risky?