What You'll Master Here
Transformations write a recipe; the action cooks it. Spark reads the whole recipe first (lazily), then runs it as efficiently as it can, splitting at the wide steps that need a shuffle.
This chapter is the hinge of the whole module. Once you truly understand transformations, actions, and lazy evaluation, the Spark UI, performance tuning, and "why did my job do that?" all make sense. It is the model that explains everything.
You will learn the two kinds of operations (transformations that build a plan, actions that run it), why Spark is lazy and what that buys you, the crucial split between narrow and wide transformations, and how the lazy plan (a DAG) becomes the jobs, stages, and tasks you met in Chapters 2 and 3.
Everything is concrete: a DataFrame chain that visibly does nothing until an action, plus diagrams of narrow vs wide and of the DAG turning into stages.
Transformations write a recipe; the action cooks it. Spark reads the whole recipe first (lazily), then runs it as efficiently as it can, splitting at the wide steps that need a shuffle.
Lazy evaluation and the narrow/wide split are the two ideas behind nearly every Spark performance question. Miss them and Spark feels random; grasp them and you can predict and tune what it does.
- transformation
- A lazy operation that returns a new DataFrame/RDD (select, filter, groupBy).
- action
- An operation that triggers execution and returns/writes a result (count, show, write).
- lazy evaluation
- Transformations are recorded into a plan; nothing runs until an action.
- narrow vs wide
- Narrow needs no shuffle (filter); wide needs a shuffle (groupBy) and starts a new stage.
Expecting each transformation to run as you write it. You misjudge where time is spent; only the action runs the chain, and the UI shows it all at once.
Assume nothing runs until an action; plan your actions deliberately.
Count the wide transformations to predict the stages.
Use .explain() to see the plan before blaming the cluster.
Spark builds a lazy plan of transformations and only executes it when an action arrives. Wide transformations (shuffles) split that plan into stages. This one sentence explains most of Spark's behaviour.
Transformations are lazy and build a DAG; actions trigger it; wide transformations (shuffles) split it into stages. This model underpins the Spark UI and all tuning.
- Name three transformations and three actions.
- Explain why Spark being lazy lets it optimise your job.