Openingloading the content…

Loadingone moment…

PySpark

Your First Spark Job & the Spark UI

SparkSession, running code in notebooks or via spark-submit, and reading the Spark UI end to end — jobs, stages, tasks, executors, storage, and what a failed job looks like.

⏱ 7 min readTopics chapter readerLevel · Foundations

01 · Overview

Start here

⏱ 5 min · Topic 1 of 9

Chapter 2 showed the parts of Spark: the driver, the executors, jobs, stages and tasks. In this chapter you run the paid-revenue job from chapter 2 for real.

Then you open the Spark UI. The Spark UI is a web page that Spark serves while your program runs. It shows every job, stage and task.

You will learn to:

Start a SparkSession and read a CSV file.
Find the Spark UI and know what each tab shows.
Read your first job in the UI: its jobs, stages and tasks.
Find a cached table, and see what an error looks like.

Every number in this chapter comes from a real run of Spark 4.2 on one laptop. On a bigger cluster, you see more tasks, but the same steps.

02 · Setup

Start a session, read a file

⏱ 5 min · Topic 2 of 9

A SparkSession is your connection to Spark. You give it your steps, and the driver runs them (chapter 2).

Here the session runs on your own machine. master("local[2]") means: run Spark inside this one program, with 2 task slots.

Create a session and read orders.csvworked example

SQL

Input data

orders.csv6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	(empty)	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .master("local[2]")
         .appName("shop")
         .getOrCreate())

orders = spark.read.csv("orders.csv", header=True, inferSchema=True)
orders.printSchema()

Result · 6 rows

printSchema() output
root
\|-- order_id: integer (nullable = true)
\|-- customer_id: integer (nullable = true)
\|-- status: string (nullable = true)
\|-- amount: double (nullable = true)
\|-- order_date: date (nullable = true)

A schema is the list of columns and their types. inferSchema=True makes Spark guess each type from the data.

To guess the types, Spark must read the file. So this line already runs work: in our run it started 2 jobs before any action.

The file is tiny, so Spark reads it as 1 partition. Chapter 2 used 3 partitions as a model. A big file gives many partitions, and one task for each.

Watch out

On a big file, inferSchema reads the data an extra time. In a real job, give the schema yourself. Chapter 6, DataFrames & Datasets: The Structured APIs, shows how.

03 · Running

Notebooks and spark-submit

⏱ 5 min · Topic 3 of 9

You can run Spark code in two common ways:

Way	Who makes the session	Used for
A notebook, such as Jupyter or Databricks	Usually the notebook. The name is spark.	Trying ideas and exploring data.
spark-submit with a .py file	Your code, with SparkSession.builder.	Jobs that run on a schedule.

spark-submit is a command that sends a program to Spark:

Run the job as a programworked example

SQL

spark-submit --master "local[2]" revenue_job.py

revenue_job.py holds the code of this chapter. Chapter 28 shows spark-submit on a real cluster.

getOrCreate() reuses a session if one already exists. So the same code works in a notebook and in spark-submit.

04 · The UI

Reading the Spark UI

⏱ 5 min · Topic 4 of 9

While the program runs, open http://localhost:4040 in a browser. If that port is busy, Spark uses the next one: 4041, 4042, and so on.

In code, spark.sparkContext.uiWebUrl gives the address.

Each tab answers one question:

Tab	The question it answers
Jobs	Which jobs ran, and did they succeed?
Stages	Which stage was slow, and how long did each task take?
SQL / DataFrame	Which plan did Spark choose for a query?
Executors	Which executors exist, and are they busy?
Storage	Which tables are cached, and how big are they?
Environment	Which settings did this run use?

The UI stops when your program stops. To see a finished job later, you need the History Server. It works only if you set spark.eventLog.enabled=true before the run.

Watch out

Before you change any setting to make a job faster, look in the UI. It shows which stage is slow. Changing settings without looking is a guess.

05 · First job

Your first job in the UI

⏱ 9 min · Topic 5 of 9

Now run the job from chapter 2. It adds up the paid orders for each customer:

Paid revenue for each customerworked example

SQL

from pyspark.sql import functions as F

revenue = (orders
           .filter(F.col("status") == "paid")
           .groupBy("customer_id")
           .agg(F.sum("amount").alias("revenue")))

revenue.show()

Result · 3 rows

customer_id	revenue
1	65.0
3	15.5
4	80.0

Asha (1) has 40.00 + 25.00. Order 103 is refunded and order 105 is pending, so neither counts. The row order can be different.

Building revenue started no job. Only show(), an action, started work. Chapter 5, Transformations, Actions & Lazy Evaluation, explains why.

Now open the Jobs tab. In our run, show() started 2 jobs:

What the Jobs tab showed

Job	Stages	Tasks	What it did
0 and 1	1 each	1 each	Read the header line, then the data to guess the types.
2	1	1	Read the file, kept paid orders, made partial sums, and wrote them for the shuffle.
3	2 (1 skipped)	1	Read the partial sums and added them up for each customer.

groupBy must bring all rows of one customer together. Moving rows between tasks like this is a shuffle, and a shuffle splits the work into two stages.

Spark ran the stage before the shuffle as job 2. Job 3 then lists that same stage again, as skipped, because its result already exists.

Why only 1 task after the shuffle

Spark plans 200 partitions after a shuffle by default. The SQL / DataFrame tab shows the final plan. Here it is, with the internal ids removed:

The final plan (simplified)worked example

SQL

revenue.explain()

AdaptiveSparkPlan isFinalPlan=true
+- HashAggregate(keys=[customer_id], functions=[sum(amount)])
   +- AQEShuffleRead coalesced
      +- Exchange hashpartitioning(customer_id, 200)
         +- HashAggregate(keys=[customer_id], functions=[partial_sum(amount)])
            +- Filter (status = paid)
               +- FileScan csv [customer_id, status, amount]

Read it from the bottom up. Exchange is the shuffle. This final plan appears after an action has run.

Adaptive Query Execution (AQE) changes the plan while the job runs. It saw only 3 small rows after the shuffle, so it joined the 200 partitions into 1. That is AQEShuffleRead coalesced, and it is why job 3 has 1 task.

AQE is on by default. Chapter 22, Adaptive Query Execution & Cost-Based Optimization, explains it.

Interview question

“A groupBy job shows 2 stages, and the second has 1 task, not 200. Why?” The groupBy needs a shuffle, which splits the stages. AQE then joined the small shuffle partitions into one.

06 · Cluster

The Executors and Storage tabs

⏱ 5 min · Topic 6 of 9

In local mode, the driver also does the work of the executors. So the Executors tab shows one row, named driver, with 2 cores for local[2]. On a cluster, you see one row for each executor.

The Storage tab lists cached tables. To cache a table is to keep it in memory after it is first read, so later actions can reuse it:

Cache orders, then fill the cacheworked example

SQL

orders.cache()
print(orders.count())
print(orders.storageLevel)

Result · 2 rows

output
6
Disk Memory Deserialized 1x Replicated

cache() only marks the table. The count() action reads the file and fills the cache.

After count(), the Storage tab shows the cached orders and their size. The storage level says Spark keeps the table in memory, and uses the disk if memory is full.

Chapter 19, Caching, Persistence & Checkpointing, explains when caching helps.

07 · Errors

What failure looks like

⏱ 5 min · Topic 7 of 9

Many errors happen before any job runs. Spark checks your steps against the schema first. A wrong column name fails at once:

A typo in a column nameworked example

SQL

orders.select("amout")

Result · 2 rows

error (shortened)
AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `amout` cannot be resolved.
Did you mean one of the following? [`amount`, `status`, `order_id`, `order_date`, `customer_id`].

The error names the wrong column and suggests the right one. No job starts, so the Jobs tab shows nothing new.

A missing file also fails before a job, with the error [PATH_NOT_FOUND] Path does not exist.

Other errors happen inside a task, while data is read or changed. Then the Jobs tab shows a failed job. Open its stage, and the failed task shows the error.

On a cluster, Spark tries a failed task again, up to 4 times by default (spark.task.maxFailures). Plain local mode does not retry.

Watch out

A job can succeed after a retry. Retried tasks in the Stages tab can be a warning sign, such as a machine that is running out of memory.

08 · Recap

Recap

⏱ 4 min · Topic 8 of 9

When you have a question about a job, open the tab that answers it:

Question	Tab
Did my job run?	Jobs
Which step was slow?	Stages
Where is the shuffle?	SQL / DataFrame
Is my table cached?	Storage
Are my executors busy?	Executors

Check yourself. Why did reading the file start jobs before any action? (Answer: inferSchema reads the data to guess the types.)

Openingloading the content…

PySpark

Your First Spark Job & the Spark UI

SparkSession, running code in notebooks or via spark-submit, and reading the Spark UI end to end — jobs, stages, tasks, executors, storage, and what a failed job looks like.

⏱ 7 min readTopics chapter readerLevel · Foundations

01 · Overview

Start here

⏱ 5 min · Topic 1 of 9

Chapter 2 showed the parts of Spark: the driver, the executors, jobs, stages and tasks. In this chapter you run the paid-revenue job from chapter 2 for real.

Then you open the Spark UI. The Spark UI is a web page that Spark serves while your program runs. It shows every job, stage and task.

You will learn to:

Start a SparkSession and read a CSV file.
Find the Spark UI and know what each tab shows.
Read your first job in the UI: its jobs, stages and tasks.
Find a cached table, and see what an error looks like.

Every number in this chapter comes from a real run of Spark 4.2 on one laptop. On a bigger cluster, you see more tasks, but the same steps.

02 · Setup

Start a session, read a file

⏱ 5 min · Topic 2 of 9

A SparkSession is your connection to Spark. You give it your steps, and the driver runs them (chapter 2).

Here the session runs on your own machine. master("local[2]") means: run Spark inside this one program, with 2 task slots.

Create a session and read orders.csvworked example

SQL

Input data

orders.csv6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	(empty)	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .master("local[2]")
         .appName("shop")
         .getOrCreate())

orders = spark.read.csv("orders.csv", header=True, inferSchema=True)
orders.printSchema()

Result · 6 rows

printSchema() output
root
\|-- order_id: integer (nullable = true)
\|-- customer_id: integer (nullable = true)
\|-- status: string (nullable = true)
\|-- amount: double (nullable = true)
\|-- order_date: date (nullable = true)

A schema is the list of columns and their types. inferSchema=True makes Spark guess each type from the data.

To guess the types, Spark must read the file. So this line already runs work: in our run it started 2 jobs before any action.

The file is tiny, so Spark reads it as 1 partition. Chapter 2 used 3 partitions as a model. A big file gives many partitions, and one task for each.

Watch out

On a big file, inferSchema reads the data an extra time. In a real job, give the schema yourself. Chapter 6, DataFrames & Datasets: The Structured APIs, shows how.

03 · Running

Notebooks and spark-submit

⏱ 5 min · Topic 3 of 9

You can run Spark code in two common ways:

Way	Who makes the session	Used for
A notebook, such as Jupyter or Databricks	Usually the notebook. The name is spark.	Trying ideas and exploring data.
spark-submit with a .py file	Your code, with SparkSession.builder.	Jobs that run on a schedule.

spark-submit is a command that sends a program to Spark:

Run the job as a programworked example

SQL

spark-submit --master "local[2]" revenue_job.py

revenue_job.py holds the code of this chapter. Chapter 28 shows spark-submit on a real cluster.

getOrCreate() reuses a session if one already exists. So the same code works in a notebook and in spark-submit.

04 · The UI

Reading the Spark UI

⏱ 5 min · Topic 4 of 9

While the program runs, open http://localhost:4040 in a browser. If that port is busy, Spark uses the next one: 4041, 4042, and so on.

In code, spark.sparkContext.uiWebUrl gives the address.

Each tab answers one question:

Tab	The question it answers
Jobs	Which jobs ran, and did they succeed?
Stages	Which stage was slow, and how long did each task take?
SQL / DataFrame	Which plan did Spark choose for a query?
Executors	Which executors exist, and are they busy?
Storage	Which tables are cached, and how big are they?
Environment	Which settings did this run use?

The UI stops when your program stops. To see a finished job later, you need the History Server. It works only if you set spark.eventLog.enabled=true before the run.

Watch out

Before you change any setting to make a job faster, look in the UI. It shows which stage is slow. Changing settings without looking is a guess.

05 · First job

Your first job in the UI

⏱ 9 min · Topic 5 of 9

Now run the job from chapter 2. It adds up the paid orders for each customer:

Paid revenue for each customerworked example

SQL

from pyspark.sql import functions as F

revenue = (orders
           .filter(F.col("status") == "paid")
           .groupBy("customer_id")
           .agg(F.sum("amount").alias("revenue")))

revenue.show()

Result · 3 rows

customer_id	revenue
1	65.0
3	15.5
4	80.0

Asha (1) has 40.00 + 25.00. Order 103 is refunded and order 105 is pending, so neither counts. The row order can be different.

Building revenue started no job. Only show(), an action, started work. Chapter 5, Transformations, Actions & Lazy Evaluation, explains why.

Now open the Jobs tab. In our run, show() started 2 jobs:

What the Jobs tab showed

Job	Stages	Tasks	What it did
0 and 1	1 each	1 each	Read the header line, then the data to guess the types.
2	1	1	Read the file, kept paid orders, made partial sums, and wrote them for the shuffle.
3	2 (1 skipped)	1	Read the partial sums and added them up for each customer.

groupBy must bring all rows of one customer together. Moving rows between tasks like this is a shuffle, and a shuffle splits the work into two stages.

Spark ran the stage before the shuffle as job 2. Job 3 then lists that same stage again, as skipped, because its result already exists.

Why only 1 task after the shuffle

Spark plans 200 partitions after a shuffle by default. The SQL / DataFrame tab shows the final plan. Here it is, with the internal ids removed:

The final plan (simplified)worked example

SQL

revenue.explain()

AdaptiveSparkPlan isFinalPlan=true
+- HashAggregate(keys=[customer_id], functions=[sum(amount)])
   +- AQEShuffleRead coalesced
      +- Exchange hashpartitioning(customer_id, 200)
         +- HashAggregate(keys=[customer_id], functions=[partial_sum(amount)])
            +- Filter (status = paid)
               +- FileScan csv [customer_id, status, amount]

Read it from the bottom up. Exchange is the shuffle. This final plan appears after an action has run.

AQE is on by default. Chapter 22, Adaptive Query Execution & Cost-Based Optimization, explains it.

Interview question

“A groupBy job shows 2 stages, and the second has 1 task, not 200. Why?” The groupBy needs a shuffle, which splits the stages. AQE then joined the small shuffle partitions into one.

06 · Cluster

The Executors and Storage tabs

⏱ 5 min · Topic 6 of 9

In local mode, the driver also does the work of the executors. So the Executors tab shows one row, named driver, with 2 cores for local[2]. On a cluster, you see one row for each executor.

The Storage tab lists cached tables. To cache a table is to keep it in memory after it is first read, so later actions can reuse it:

Cache orders, then fill the cacheworked example

SQL

orders.cache()
print(orders.count())
print(orders.storageLevel)

Result · 2 rows

output
6
Disk Memory Deserialized 1x Replicated

cache() only marks the table. The count() action reads the file and fills the cache.

After count(), the Storage tab shows the cached orders and their size. The storage level says Spark keeps the table in memory, and uses the disk if memory is full.

Chapter 19, Caching, Persistence & Checkpointing, explains when caching helps.

07 · Errors

What failure looks like

⏱ 5 min · Topic 7 of 9

Many errors happen before any job runs. Spark checks your steps against the schema first. A wrong column name fails at once:

A typo in a column nameworked example

SQL

orders.select("amout")

Result · 2 rows

error (shortened)
AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `amout` cannot be resolved.
Did you mean one of the following? [`amount`, `status`, `order_id`, `order_date`, `customer_id`].

The error names the wrong column and suggests the right one. No job starts, so the Jobs tab shows nothing new.

A missing file also fails before a job, with the error [PATH_NOT_FOUND] Path does not exist.

Other errors happen inside a task, while data is read or changed. Then the Jobs tab shows a failed job. Open its stage, and the failed task shows the error.

On a cluster, Spark tries a failed task again, up to 4 times by default (spark.task.maxFailures). Plain local mode does not retry.

Watch out

A job can succeed after a retry. Retried tasks in the Stages tab can be a warning sign, such as a machine that is running out of memory.

08 · Recap

Recap

⏱ 4 min · Topic 8 of 9

When you have a question about a job, open the tab that answers it:

Question	Tab
Did my job run?	Jobs
Which step was slow?	Stages
Where is the shuffle?	SQL / DataFrame
Is my table cached?	Storage
Are my executors busy?	Executors

Check yourself. Why did reading the file start jobs before any action? (Answer: inferSchema reads the data to guess the types.)

Your First Spark Job & the Spark UI

Start here

Start a session, read a file

Notebooks and spark-submit

Reading the Spark UI

Your first job in the UI

Why only 1 task after the shuffle

The Executors and Storage tabs

What failure looks like

Recap

Next Chapter

Your First Spark Job & the Spark UI

Start here

Start a session, read a file

Notebooks and spark-submit

Reading the Spark UI

Your first job in the UI

Why only 1 task after the shuffle

The Executors and Storage tabs

What failure looks like

Recap

Next Chapter