Openingloading the content…

Loadingone moment…

PySpark

Caching, Persistence & Checkpointing

cache and persist, storage levels, when caching helps or hurts, and truncating long lineage with checkpointing.

⏱ 9 min readTopics chapter readerLevel · Performance

01 · Orientation

Start here

⏱ 5 min · Topic 1 of 11

Spark is lazy. Each action starts its own job, and each job runs the whole plan again. If two actions use the same DataFrame, Spark computes it twice.

You can tell Spark to keep a result in memory, so the second action does not start over. This is called caching.

You can also cut a long plan and save the result to reliable storage. This is called checkpointing.

This chapter has three parts.

The problem: two actions that read the same data twice.
Caching: how to keep the data, which storage level to use, and when not to cache.
Lineage and checkpoints: what to do when a plan grows too long.

We use the shop again. We work with the 6 orders.

02 · The problem

Two actions, two reads

⏱ 6 min · Topic 2 of 11

You saw this in the chapter "Transformations, Actions & Lazy Evaluation": two actions read the source twice. Here is the same problem again.

First you count the paid orders. Then you add up the amount for each customer.

Two actions on the same DataFrameworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

paid = orders.filter("status = 'paid'")

paid.count()

paid.groupBy("customer_id").sum("amount").show()

count() returns 4. show() prints (3 rows)

customer_id	sum(amount)
1	65.00
3	15.50
4	80.00

Four orders are paid: 101, 102, 104 and 106. Customer 1 has 40.00 + 25.00 = 65.00. Row order can change from run to run.

The filter is a transformation, so it runs nothing. Each action starts a job.

Now look at what each job did.

Job	Action	Reads the source files?
Job 0	count()	Yes
Job 1	show()	Yes, again

Spark numbers jobs from 0 in the Spark UI. One action can start more than one job. We show one job for each action to keep it simple.

Job 1 did not know that job 0 had just read and filtered the same rows. It started from the source.

With 6 rows this costs nothing. With 600 million rows, the second read is real work.

03 · Keep the result

cache() and reuse

⏱ 6 min · Topic 3 of 11

cache() tells Spark to keep a DataFrame, so later actions can reuse it. Spark keeps it in memory first, and on disk if memory is full. cache() is lazy, so calling it runs nothing.

Spark fills the cache during the first action.

Cache the paid ordersworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

paid = orders.filter("status = 'paid'").cache()

paid.count()

paid.groupBy("customer_id").sum("amount").show()

count() returns 4. show() prints (3 rows)

customer_id	sum(amount)
1	65.00
3	15.50
4	80.00

The output is the same as before. Only the work behind it changed.

The first line only asks Spark to cache paid. It stores nothing yet. The count() fills the cache with the 4 paid rows. The show() reads them from memory.

Compare the two jobs with the first example.

Job	Action	Reads the source files?	What it does
Job 0	count()	Yes	Reads, filters, and stores the 4 rows in memory
Job 1	show()	No	Reads the 4 rows from memory

The first action still does all the work. Only the actions after it save work. So a cache helps only when at least two actions use the DataFrame.

Watch out

cache() alone stores nothing. If no action runs, the cache stays empty. Add an action, such as count(), when you want the cache ready.

04 · Where to keep it

Storage levels with persist()

⏱ 4 min · Topic 4 of 11

A storage level says where Spark keeps the cached data. persist() lets you choose the level. cache() on a DataFrame uses MEMORY_AND_DISK.

Choose a levelworked example

SQL

from pyspark import StorageLevel

# paid is not cached yet
paid = orders.filter("status = 'paid'").persist(StorageLevel.DISK_ONLY)

Three storage levels (reference)

Level	Where it keeps the data	If memory is too small
MEMORY_ONLY	Memory only	Partitions that do not fit are not stored. Spark computes them again when needed.
MEMORY_AND_DISK	Memory first, then disk	Partitions that do not fit go to disk.
DISK_ONLY	Disk only	Not a problem, but reads are slower than memory.

Spark has more levels, such as copies on two executors. These three are enough to start.

Use the default unless you have a reason to change it. Newer Spark versions may show the default as MEMORY_AND_DISK_DESER. It means the same thing: memory first, then disk.

Choose the level before the first action. If the DataFrame is already cached, persist() with a new level changes nothing. Call unpersist() first.

In the chapter "Spark Architecture: Driver, Executors & Cluster Managers", executor memory has two parts. Cached data lives in storage memory. Execution memory is for work like joins, grouping and sorting.

05 · Clean up and check

unpersist() and the Storage tab

⏱ 5 min · Topic 5 of 11

When you finish with a cached DataFrame, remove it. unpersist() frees the memory and the disk space.

Release the cacheworked example

SQL

paid.unpersist()

Effect

Before unpersist()	After unpersist()
4 paid rows kept in memory	Nothing kept. The next action reads the source again.

Later actions on paid still work. They just compute it again.

To see what is cached, open the Spark UI and click the Storage tab. It has one row for each cached DataFrame.

Each row has these columns: Storage Level, Cached Partitions, Fraction Cached, Size in Memory and Size on Disk. If Fraction Cached is below 100%, some partitions are not cached yet. For example, show() may read only the first partitions.

If a DataFrame is missing from the Storage tab, no action has filled it yet. Or you already called unpersist().

06 · Good judgment

When not to cache

⏱ 5 min · Topic 6 of 11

A cache costs memory. Executors need that memory for joins and shuffles. Do not cache in these three cases.

The DataFrame is used by only one action. Nothing reads the cache, so you only pay.
The DataFrame is much bigger than storage memory. Most partitions go to disk, and reading them can cost as much as reading the source.
The DataFrame is cheap to compute. Reading a small file again can cost less than the cache.

A common mistake is to cache too early. Compare two ways to cache the paid orders.

Cache before or after the filterworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

# Too early
orders.cache()
paid = orders.filter("status = 'paid'")

# Better
paid = orders.filter("status = 'paid'").cache()

Rows kept in memory

Where you call cache()	Rows cached	Rows your later actions use
Before the filter	6	4
After the filter	4	4

Orders 103 (refunded) and 105 (pending) are never used, but the first version keeps them.

Cache the DataFrame that your later actions really read. Here that is paid.

Six rows against four is small. With a big table, the difference is a lot of memory that other work needs.

07 · The plan Spark remembers

Lineage grows in loops

⏱ 5 min · Topic 7 of 11

Spark remembers how it built each DataFrame. This chain of steps is the lineage. If a partition is lost, Spark follows the lineage to build it again.

A loop adds steps every round. Look at this loop, and then at how the lineage grows.

A loop that adds a step each roundworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import functions as F

df = orders
for i in range(3):
    df = df.withColumn("amount", F.col("amount") * 1.1)

Lineage after each round

After round	Steps Spark remembers
Before the loop	read orders
1	read orders, step 1
2	read orders, step 1, step 2
3	read orders, step 1, step 2, step 3

Nothing has run yet. These are only plan steps.

After 3 rounds the plan has 3 steps. After 50 rounds it has 50.

A long lineage makes the plan slow to build. It also means a lost partition is slow to rebuild, because Spark replays every step.

A cache does not fix this. A cached DataFrame still keeps its lineage. You need a checkpoint.

08 · Cut the plan

Checkpoint cuts lineage

⏱ 7 min · Topic 8 of 11

A checkpoint saves the data to files and forgets the steps before it. After that, the lineage starts at the saved files. First you set the folder where Spark saves them.

Checkpoint inside a loopworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import functions as F

spark.sparkContext.setCheckpointDir("s3://shop/checkpoints")

df = orders
for i in range(50):
    df = df.withColumn("amount", F.col("amount") * 1.1)
    if i % 10 == 9:
        df = df.checkpoint()

Lineage at some loop steps (i is the loop counter)

Loop step	Steps Spark remembers
i = 9, after checkpoint()	read the checkpoint files
i = 10	read the checkpoint files, step 1
i = 18	read the checkpoint files, step 1 to step 9
i = 19, after checkpoint()	read the checkpoint files

The list starts again at the files after every checkpoint.

This loop checkpoints every 10 rounds, so the plan never holds more than 10 steps after the files.

For a DataFrame, checkpoint() is eager by default. It runs a job right away to write the files.

This is different from cache(), which is lazy.

There is also localCheckpoint(). It cuts the lineage too, but it keeps the data on the executors, not in reliable storage.

If an executor is lost, the data is lost. It skips the write to the checkpoint folder, but it is less safe.

Spark does not delete checkpoint files by default. Clean up the folder when the job is done.

	cache()	checkpoint()	localCheckpoint()
Where the data goes	Executor memory or disk	Checkpoint folder	Executor storage
Cuts the lineage?	No	Yes	Yes
Safe if an executor is lost?	Spark can rebuild it from the lineage	Yes	No

Interview question

The question: "When do you use checkpoint instead of cache?" A good answer is short. "When the lineage is long, for example in a loop. Checkpoint writes the data to reliable storage and cuts the lineage."

Openingloading the content…

PySpark

Caching, Persistence & Checkpointing

cache and persist, storage levels, when caching helps or hurts, and truncating long lineage with checkpointing.

⏱ 9 min readTopics chapter readerLevel · Performance

01 · Orientation

Start here

⏱ 5 min · Topic 1 of 11

Spark is lazy. Each action starts its own job, and each job runs the whole plan again. If two actions use the same DataFrame, Spark computes it twice.

You can tell Spark to keep a result in memory, so the second action does not start over. This is called caching.

You can also cut a long plan and save the result to reliable storage. This is called checkpointing.

This chapter has three parts.

The problem: two actions that read the same data twice.
Caching: how to keep the data, which storage level to use, and when not to cache.
Lineage and checkpoints: what to do when a plan grows too long.

We use the shop again. We work with the 6 orders.

02 · The problem

Two actions, two reads

⏱ 6 min · Topic 2 of 11

You saw this in the chapter "Transformations, Actions & Lazy Evaluation": two actions read the source twice. Here is the same problem again.

First you count the paid orders. Then you add up the amount for each customer.

Two actions on the same DataFrameworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

paid = orders.filter("status = 'paid'")

paid.count()

paid.groupBy("customer_id").sum("amount").show()

count() returns 4. show() prints (3 rows)

customer_id	sum(amount)
1	65.00
3	15.50
4	80.00

Four orders are paid: 101, 102, 104 and 106. Customer 1 has 40.00 + 25.00 = 65.00. Row order can change from run to run.

The filter is a transformation, so it runs nothing. Each action starts a job.

Now look at what each job did.

Job	Action	Reads the source files?
Job 0	count()	Yes
Job 1	show()	Yes, again

Spark numbers jobs from 0 in the Spark UI. One action can start more than one job. We show one job for each action to keep it simple.

Job 1 did not know that job 0 had just read and filtered the same rows. It started from the source.

With 6 rows this costs nothing. With 600 million rows, the second read is real work.

03 · Keep the result

cache() and reuse

⏱ 6 min · Topic 3 of 11

cache() tells Spark to keep a DataFrame, so later actions can reuse it. Spark keeps it in memory first, and on disk if memory is full. cache() is lazy, so calling it runs nothing.

Spark fills the cache during the first action.

Cache the paid ordersworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

paid = orders.filter("status = 'paid'").cache()

paid.count()

paid.groupBy("customer_id").sum("amount").show()

count() returns 4. show() prints (3 rows)

customer_id	sum(amount)
1	65.00
3	15.50
4	80.00

The output is the same as before. Only the work behind it changed.

The first line only asks Spark to cache paid. It stores nothing yet. The count() fills the cache with the 4 paid rows. The show() reads them from memory.

Compare the two jobs with the first example.

Job	Action	Reads the source files?	What it does
Job 0	count()	Yes	Reads, filters, and stores the 4 rows in memory
Job 1	show()	No	Reads the 4 rows from memory

The first action still does all the work. Only the actions after it save work. So a cache helps only when at least two actions use the DataFrame.

Watch out

cache() alone stores nothing. If no action runs, the cache stays empty. Add an action, such as count(), when you want the cache ready.

04 · Where to keep it

Storage levels with persist()

⏱ 4 min · Topic 4 of 11

A storage level says where Spark keeps the cached data. persist() lets you choose the level. cache() on a DataFrame uses MEMORY_AND_DISK.

Choose a levelworked example

SQL

from pyspark import StorageLevel

# paid is not cached yet
paid = orders.filter("status = 'paid'").persist(StorageLevel.DISK_ONLY)

Three storage levels (reference)

Level	Where it keeps the data	If memory is too small
MEMORY_ONLY	Memory only	Partitions that do not fit are not stored. Spark computes them again when needed.
MEMORY_AND_DISK	Memory first, then disk	Partitions that do not fit go to disk.
DISK_ONLY	Disk only	Not a problem, but reads are slower than memory.

Spark has more levels, such as copies on two executors. These three are enough to start.

Use the default unless you have a reason to change it. Newer Spark versions may show the default as MEMORY_AND_DISK_DESER. It means the same thing: memory first, then disk.

Choose the level before the first action. If the DataFrame is already cached, persist() with a new level changes nothing. Call unpersist() first.

05 · Clean up and check

unpersist() and the Storage tab

⏱ 5 min · Topic 5 of 11

When you finish with a cached DataFrame, remove it. unpersist() frees the memory and the disk space.

Release the cacheworked example

SQL

paid.unpersist()

Effect

Before unpersist()	After unpersist()
4 paid rows kept in memory	Nothing kept. The next action reads the source again.

Later actions on paid still work. They just compute it again.

To see what is cached, open the Spark UI and click the Storage tab. It has one row for each cached DataFrame.

If a DataFrame is missing from the Storage tab, no action has filled it yet. Or you already called unpersist().

06 · Good judgment

When not to cache

⏱ 5 min · Topic 6 of 11

A cache costs memory. Executors need that memory for joins and shuffles. Do not cache in these three cases.

The DataFrame is used by only one action. Nothing reads the cache, so you only pay.
The DataFrame is much bigger than storage memory. Most partitions go to disk, and reading them can cost as much as reading the source.
The DataFrame is cheap to compute. Reading a small file again can cost less than the cache.

A common mistake is to cache too early. Compare two ways to cache the paid orders.

Cache before or after the filterworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

# Too early
orders.cache()
paid = orders.filter("status = 'paid'")

# Better
paid = orders.filter("status = 'paid'").cache()

Rows kept in memory

Where you call cache()	Rows cached	Rows your later actions use
Before the filter	6	4
After the filter	4	4

Orders 103 (refunded) and 105 (pending) are never used, but the first version keeps them.

Cache the DataFrame that your later actions really read. Here that is paid.

Six rows against four is small. With a big table, the difference is a lot of memory that other work needs.

07 · The plan Spark remembers

Lineage grows in loops

⏱ 5 min · Topic 7 of 11

Spark remembers how it built each DataFrame. This chain of steps is the lineage. If a partition is lost, Spark follows the lineage to build it again.

A loop adds steps every round. Look at this loop, and then at how the lineage grows.

A loop that adds a step each roundworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import functions as F

df = orders
for i in range(3):
    df = df.withColumn("amount", F.col("amount") * 1.1)

Lineage after each round

After round	Steps Spark remembers
Before the loop	read orders
1	read orders, step 1
2	read orders, step 1, step 2
3	read orders, step 1, step 2, step 3

Nothing has run yet. These are only plan steps.

After 3 rounds the plan has 3 steps. After 50 rounds it has 50.

A long lineage makes the plan slow to build. It also means a lost partition is slow to rebuild, because Spark replays every step.

A cache does not fix this. A cached DataFrame still keeps its lineage. You need a checkpoint.

08 · Cut the plan

Checkpoint cuts lineage

⏱ 7 min · Topic 8 of 11

A checkpoint saves the data to files and forgets the steps before it. After that, the lineage starts at the saved files. First you set the folder where Spark saves them.

Checkpoint inside a loopworked example

SQL

Input data

orders6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

from pyspark.sql import functions as F

spark.sparkContext.setCheckpointDir("s3://shop/checkpoints")

df = orders
for i in range(50):
    df = df.withColumn("amount", F.col("amount") * 1.1)
    if i % 10 == 9:
        df = df.checkpoint()

Lineage at some loop steps (i is the loop counter)

Loop step	Steps Spark remembers
i = 9, after checkpoint()	read the checkpoint files
i = 10	read the checkpoint files, step 1
i = 18	read the checkpoint files, step 1 to step 9
i = 19, after checkpoint()	read the checkpoint files

The list starts again at the files after every checkpoint.

This loop checkpoints every 10 rounds, so the plan never holds more than 10 steps after the files.

For a DataFrame, checkpoint() is eager by default. It runs a job right away to write the files.

This is different from cache(), which is lazy.

There is also localCheckpoint(). It cuts the lineage too, but it keeps the data on the executors, not in reliable storage.

If an executor is lost, the data is lost. It skips the write to the checkpoint folder, but it is less safe.

Spark does not delete checkpoint files by default. Clean up the folder when the job is done.

	cache()	checkpoint()	localCheckpoint()
Where the data goes	Executor memory or disk	Checkpoint folder	Executor storage
Cuts the lineage?	No	Yes	Yes
Safe if an executor is lost?	Spark can rebuild it from the lineage	Yes	No

Interview question

Caching, Persistence & Checkpointing

Start here

Two actions, two reads

cache() and reuse

Storage levels with persist()

unpersist() and the Storage tab

When not to cache

Lineage grows in loops

Checkpoint cuts lineage

Shuffle files are reused

Check yourself

Next Chapter

Caching, Persistence & Checkpointing

Start here

Two actions, two reads

cache() and reuse

Storage levels with persist()

unpersist() and the Storage tab

When not to cache

Lineage grows in loops

Checkpoint cuts lineage

Shuffle files are reused

Check yourself

Next Chapter