What You'll Master Here
Application = your whole program (one SparkSession). Action → job. Shuffle → new stage. Partition → task. Four nested levels, each with one trigger.
You have seen the pieces: driver and executors (Chapter 2), lazy DAGs (Chapter 5), Catalyst (Chapter 13). This chapter assembles them into one clear story: what actually happens, end to end, when you call an action on a cluster.
The hierarchy is the whole chapter: an application contains jobs, a job contains stages, a stage contains tasks. Each level has a precise trigger. An action starts a job; a shuffle boundary splits a job into stages; the number of partitions sets the number of tasks. Get this hierarchy and the Spark UI stops being mysterious.
With a diagram of the application → job → stage → task hierarchy, the role of the DAG scheduler, and exactly where stage boundaries come from.
Application = your whole program (one SparkSession). Action → job. Shuffle → new stage. Partition → task. Four nested levels, each with one trigger.
Almost every performance and debugging conversation uses this vocabulary. "Which stage is slow? How many tasks? Why two stages?" If the hierarchy is second nature, the Spark UI and tuning all click into place.
- application
- Your whole Spark program, one SparkSession with its driver and executors.
- job
- All the work triggered by a single action (e.g., count, write, collect).
- stage
- A set of tasks that run without a shuffle; stages are split by shuffle boundaries.
- task
- The smallest unit of work: one stage's computation on one partition.
Confusing jobs, stages, and tasks. You cannot read the Spark UI or reason about performance; every tuning discussion uses these terms.
Map any slowness to a level: which job, which stage, which tasks.
Expect one job per action, and a new stage at every shuffle.
Remember: number of tasks in a stage = number of partitions.
Application → job (per action) → stage (split by shuffles) → task (per partition). Four levels, three triggers. This single hierarchy explains the entire Spark UI.
A Spark application runs jobs (one per action), each split into stages at shuffle boundaries, each stage running one task per partition. This hierarchy is the foundation for reading the UI and tuning.
- What triggers a job? What triggers a new stage? What sets the task count?
- Define application, job, stage, and task in one sentence each.