What You'll Master Here
execution memory is the workbench (active processing), storage memory is the shelf (cached data). They flex into each other, and if the workbench overflows, work spills onto the floor (disk).
Spark is an in-memory engine, so understanding how it uses executor memory is the key to avoiding its two most feared failures: OutOfMemoryError (the job crashes) and excessive spill (the job crawls). This chapter opens the executor memory model and the Tungsten engine that makes Spark fast.
You will learn how an executor divides its heap into execution memory (for shuffles, joins, aggregations, sorts) and storage memory (for cached data), how they share space dynamically, what off-heap memory is, and how Tungsten, binary memory layout plus whole-stage code generation, gives the structured APIs their speed.
With a diagram of the memory regions and a worked OOM/spill diagnosis. This is the chapter that turns cryptic memory errors into a checklist.
An executor's memory is a shared workshop: execution memory is the workbench (active processing), storage memory is the shelf (cached data). They flex into each other, and if the workbench overflows, work spills onto the floor (disk).
OOM and spill are the most common ways Spark jobs fail or slow down. Knowing the memory regions and how to tune them (executor memory, partition size, memory fraction) is essential operational skill, and a frequent interview deep-dive.
- execution memory
- Memory for shuffles, joins, sorts, and aggregations during processing.
- storage memory
- Memory for cached/persisted data blocks.
- spill
- Writing data to disk when it does not fit in execution memory.
- Tungsten
- Spark's engine for binary memory layout and whole-stage code generation.
Responding to every OOM by just raising executor memory. Often the real fix is smaller partitions or less skew; throwing memory at it is costly and may not help.
Read OOM/spill as a signal about partition size and skew, not just total memory.
Know the split: execution vs storage, sharing one unified region.
Trust Tungsten codegen; write clear structured-API code so it can apply.
Executor memory is shared between active processing (execution) and cached data (storage). Most memory failures are really partition-size or skew problems. Tungsten is why the structured APIs are fast: compact binary data and generated code.
Executor memory splits into execution and storage regions that flex dynamically; OOM and spill usually trace to oversized partitions or skew, not just low total memory. Tungsten (binary layout + codegen) powers the structured APIs' speed.
- What are the two main regions of executor memory, and what does each hold?
- Why is raising executor memory not always the right fix for OOM?