What You'll Master Here
The Spark UI is the X-ray of your job. The driver builds jobs, stages, and tasks (Chapter 2); the UI shows you all of them, plus how long each took and where the time went. An action triggers at least one job, and each job is your ticket into the UI.
You now know what Spark is (Chapter 1) and how it runs on a cluster (Chapter 2). This chapter puts you at the keyboard: you will create a SparkSession, run a real job, and learn to read the Spark UI — the dashboard where every concept from Chapter 2 appears as something you can click and inspect.
The Spark UI is the single most useful debugging and tuning tool you have. Engineers who can read it diagnose slow jobs in minutes; those who cannot guess. We will run one small job and trace it end to end through the UI: from the code you wrote, to the job the driver created, to the stages and tasks that executors ran.
Everything is concrete: the code, the input data, the output, and exactly which UI tab shows what. By the end you will be able to predict what the UI will show before you run — and confirm it afterward.
The Spark UI is the X-ray of your job. The driver builds jobs, stages, and tasks (Chapter 2); the UI shows you all of them, plus how long each took and where the time went. An action triggers at least one job, and each job is your ticket into the UI.
Almost every "why is my Spark job slow?" answer lives in the Spark UI — the number of stages, the task durations, the shuffle sizes, the skew. Learning to read it now pays off in every later tuning chapter and every on-call incident.
- SparkSession
- Your entry point to Spark: you create one, then use it to read data and run jobs.
- Spark UI
- A web dashboard (usually port 4040) showing jobs, stages, tasks, storage, and SQL plans for a running application.
- action
- An operation (show, count, write) that triggers execution and creates one or more jobs.
- job lifecycle
- The path from your code to a result: action fires → job created → split into stages → each stage runs tasks → each UI tab tracks one layer.
show() / count()
Jobs tab
Job (id 0, 1, 2…)
Jobs tab
Stage 0, Stage 1
Stages tab
one per partition
Stages → task list
SQL / DataFrame tab — shows the plan (FileScan → Exchange → Aggregate) across all stages
Treating the Spark UI as optional. You end up guessing at performance problems instead of reading the exact stage and task that is slow.
Assuming exactly one action always produces exactly one job. Some operations (inferSchema, AQE shuffle-stat sampling) fire extra jobs at read time. The Jobs tab can show two or three jobs for one show() call — this is normal.
Open the Spark UI for every non-trivial job and glance at Jobs, Stages, and the SQL tab.
Create exactly one SparkSession per application and reuse it.
Trigger work with small actions (show, limit) while exploring, not collect() on big data.
Writing Spark code is half the job; reading the Spark UI is the other half. The UI turns the abstract job/stage/task model into concrete numbers you can act on.
Locally and on most platforms the UI is at http://localhost:4040 while a job runs. If port 4040 is busy (another Spark app is running), Spark increments to :4041, :4042, and so on — check the driver log for the actual port. On Databricks/EMR the UI is linked from the cluster or job page.
A SparkSession is your handle to the cluster; an action triggers at least one job; and the Spark UI shows those jobs as jobs, stages, and tasks you can inspect. Reading the UI is the core operational skill this chapter builds.
- What object do you create first in every Spark program, and what do you use it for?
- Name two things the Spark UI can tell you that the code alone cannot.
- Why might you see two or three jobs in the UI after a single show() call?