What You'll Master Here
it gives the manager the workers in the first place.
In Chapter 1 you learned that Spark splits big data across a cluster and processes it in parallel. This chapter opens the hood: who actually does that work, and how your few lines of PySpark turn into thousands of small tasks running across many machines.
There are only a handful of moving parts, and once you can name them, the Spark UI, performance tuning, and most interview questions stop being mysterious. We will meet the driver and the executors, follow a job as it becomes stages and tasks, do the simple math that decides how much runs in parallel, and see how cluster managers (standalone, YARN, Kubernetes) and deploy modes fit in.
Everything here is concrete: a real PySpark job, the stages and tasks Spark builds from it, and the arithmetic that tells you how many tasks run at once.
The driver is the manager who plans the work and hands out assignments. Executors are the workers who do the assignments in parallel. The cluster manager is HR: it gives the manager the workers in the first place.
You cannot tune, debug, or reason about cost in Spark without this model. "My job is slow", "the driver crashed", "why 200 tasks?", all of these are answered by knowing the driver/executor/job/stage/task picture.
- driver
- The process that runs your main program, plans the work, and coordinates everything.
- executor
- A worker process on a cluster node that runs tasks and holds cached data.
- job
- All the work triggered by one action (like write() or count()).
- stage
- A group of tasks that can run without a shuffle; stages are split at shuffle boundaries.
- task
- The smallest unit of work: one stage running on one partition of data.
Picturing Spark as one program on one machine. You misread the Spark UI and cannot explain parallelism, the driver/executor split is the whole point.
Name the part you mean: driver, executor, job, stage, or task, not just "Spark".
When a job is slow, ask which stage and how many tasks, not just "why is Spark slow".
Keep heavy data on the executors; keep the driver light.
One driver plans and coordinates; many executors do the work in parallel. Your DataFrame code becomes a plan (a DAG) that the driver breaks into stages, and each stage into one task per partition.
Every term here, jobs, stages, tasks, executors, is a literal tab or column in the Spark UI (covered next chapter). Learn the words now and the UI reads like a sentence later.
Spark runs as one coordinating driver plus many parallel executors; your code becomes a job, split into stages at shuffle boundaries, split into one task per partition.
- Define driver, executor, job, stage, and task in one line each.
- Use the "manager, workers, HR" analogy to describe a Spark cluster.