What You'll Master Here
Without caching, every action replays the full recipe from raw ingredients. cache() saves the cooked dish so the next person just reheats it. Worth it only if the dish is reused.
Because transformations are lazy (Chapter 5), Spark recomputes a DataFrame from scratch every time you trigger an action on it. If you reuse the same DataFrame several times, that is wasteful, you recompute the whole lineage each time. Caching fixes this: it stores the computed result so later actions reuse it instead of recomputing.
This chapter covers cache() and persist(), the storage levels (memory, disk, serialized), when caching genuinely helps versus when it hurts (it is not free, it consumes memory), and checkpointing, a different tool that truncates a long lineage by writing it to reliable storage.
With a diagram of recompute-vs-cache and a worked example of caching a reused DataFrame. The key judgment: cache only what you reuse, and remember to unpersist.
Without caching, every action replays the full recipe from raw ingredients. cache() saves the cooked dish so the next person just reheats it. Worth it only if the dish is reused.
Caching the right DataFrame can turn a job that recomputes an expensive pipeline five times into one that computes it once. Caching the wrong thing wastes memory and can slow the job down, knowing the difference matters.
- cache()
- Marks a DataFrame to be stored after first computation (memory by default).
- persist(level)
- Like cache but with an explicit storage level (memory/disk/serialized).
- storage level
- Where and how cached data is kept: MEMORY_ONLY, MEMORY_AND_DISK, etc.
- checkpointing
- Writing a DataFrame to reliable storage to cut (truncate) its lineage.
Caching a DataFrame that is used only once. You pay the memory cost (and eviction of other data) for zero reuse benefit, a net loss.
Cache only DataFrames you reuse across multiple actions.
Pick the storage level deliberately; the default is MEMORY_AND_DISK for DataFrames.
unpersist() when done to free memory for the rest of the job.
Laziness means recompute-by-default. Caching trades memory for avoided recomputation, worth it precisely when a DataFrame is reused. Checkpointing instead trades a write for a shorter, safer lineage.
Cache a DataFrame when you reuse it across actions, to avoid recomputing its lineage each time. Choose the storage level deliberately, unpersist when done, and use checkpointing to truncate overly long lineages.
- Why does Spark recompute a DataFrame on every action by default?
- When is caching a net loss rather than a win?