Resilient Distributed Datasets: partitions, lineage, fault tolerance, and when low-level RDDs still matter under the modern APIs.
⏱ 12 min readTopics chapter readerLevel · Foundations
01 · Orientation
What You'll Master Here
An RDD is a single logical collection cut into partitions, each living on a worker. It remembers how it was built (its lineage), so any lost partition can be recomputed instead of restored from a backup.
⏱ 4 min · Topic 1 of 6
Underneath every DataFrame, every Spark SQL query, and every streaming job sits one foundation: the RDD, the Resilient Distributed Dataset. You will rarely write RDD code today, but understanding RDDs is what makes partitions, lineage, and fault tolerance click, and those concepts power everything above them.
This chapter explains what an RDD actually is (a partitioned, immutable collection spread across the cluster), how Spark recovers from machine failure for free using lineage, and why we almost always prefer DataFrames now, while still standing on RDDs underneath.
We keep it concrete with a tiny RDD example you can run, plus diagrams of how an RDD is partitioned and how a lost partition is recomputed.
Core mental model
An RDD is a single logical collection cut into partitions, each living on a worker. It remembers how it was built (its lineage), so any lost partition can be recomputed instead of restored from a backup.
Why it matters
Partitions, lineage, and fault tolerance are RDD ideas that resurface in every later chapter and many interviews ("how does Spark recover from a failed node?"). Learn them here once and the rest of Spark rests on solid ground.
RDD
Resilient Distributed Dataset: an immutable, partitioned collection processed in parallel.
partition
One slice of an RDD; each is processed by one task on one core.
lineage
The recorded chain of transformations that built an RDD; used to recompute lost data.
immutable
RDDs never change in place; a transformation creates a new RDD.
Common mistake
Reaching for RDDs for everyday data work. You lose Catalyst optimization and write more code; prefer DataFrames unless you truly need low-level control.
Better habit
Think of any dataset as "partitions across the cluster", that is the RDD view.
Explain fault tolerance via lineage (recompute), not backups.
Default to DataFrames; treat RDDs as the foundation, not the daily tool.
The big idea
An RDD = partitions + lineage. Partitions give parallelism; lineage gives fault tolerance. DataFrames add a schema and an optimizer on top, but inherit both.
Remember this
An RDD is an immutable, partitioned, fault-tolerant collection. Partitions enable parallelism; lineage enables recompute-on-failure. DataFrames are built on this foundation.
Practice2 prompts
Define an RDD in one sentence using "partitions" and "lineage".
Why do we prefer DataFrames day to day if RDDs are the foundation?
02 · The abstraction
What an RDD Actually Is
One RDD = many partitions. Picture a deck of cards dealt into piles across the table; each player (executor) works their pile at the same time.
⏱ 5 min · Topic 2 of 6
An RDD is one logical collection of items, but physically it is split into partitions, and each partition lives on a worker node in the cluster. When Spark runs a transformation, it runs it on every partition in parallel, one task per partition. That is where the "distributed" and the parallelism come from.
Two more properties matter. RDDs are immutable: you never modify one in place; each transformation (map, filter) produces a new RDD. And RDDs are lazy: transformations just record what to do; nothing runs until an action (like collect or count) is called, the same lazy model you will see for DataFrames in Chapter 5.
The diagram shows the idea: a single RDD of six numbers, cut into three partitions across three executors, ready to be processed at the same time.
Core mental model
One RDD = many partitions. Picture a deck of cards dealt into piles across the table; each player (executor) works their pile at the same time.
Why it matters
Almost every performance conversation in Spark, parallelism, skew, shuffles, comes back to "how is this partitioned?". The RDD partition model is where that intuition starts.
sparkContext (sc)
The lower-level entry point for RDDs; available as spark.sparkContext.
parallelize()
Creates an RDD from a local collection, splitting it into partitions.
lazy evaluation
Transformations are recorded, not run, until an action triggers them.
number of partitions
Controls parallelism; check it with rdd.getNumPartitions().
ONE RDD (logical)
numbers = [1, 2, 3, 4, 5, 6]
immutable · processed in parallel
↓ split into partitions ↓
EXECUTOR 1
Partition 0
12
EXECUTOR 2
Partition 1
34
EXECUTOR 3
Partition 2
56
An RDD is one logical collection split into partitions, each living on a worker and processed by its own task. More partitions (up to a point) means more parallelism. Real data rarely divides this cleanly — uneven partition sizes (data skew) are a common performance problem covered in the data-skew-and-salting chapter.
Common mistake
Assuming an RDD is one blob on one machine. You miss that it is many partitions across nodes, which is the whole basis of parallelism.
Better habit
Check getNumPartitions() to understand a dataset's parallelism.
Remember each transformation returns a new (immutable) RDD.
Expect nothing to run until an action, RDDs are lazy.
See the partitions
rdd.getNumPartitions() tells you how many pieces your data is in, and therefore how many tasks a stage over it will run.
Remember this
An RDD is one logical, immutable collection split into partitions across workers and processed in parallel, lazily. Partition count is your parallelism dial.
Practice2 prompts
What does rdd.getNumPartitions() tell you, and why does it matter?
Explain "immutable" and "lazy" in the context of RDD transformations.
03 · Resilience
Lineage & Fault Tolerance
Lineage is a recipe, not a photograph. Lose a dish (partition) and you cook it again from the recipe, rather than restoring a saved copy. But if the ingredient (source data) has expired, even the recipe cannot save you — that is what checkpointing to reliable storage is for.
⏱ 6 min · Topic 3 of 6
The "Resilient" in RDD comes from lineage. Instead of copying data around for safety, Spark remembers the exact sequence of transformations that built each RDD, a recipe. If a machine dies and takes some partitions with it, Spark just re-runs the recipe for those partitions on another machine. No backup needed. This is the key distinction: RDD fault tolerance = recompute from lineage, NOT data replication. Replication is the storage layer's job (HDFS/S3); Spark handles compute recovery.
This is why RDDs (and the DataFrames built on them) are fault-tolerant by design. A failed task or lost executor does not fail the job; the driver recomputes the missing partitions from lineage and carries on. It is also why the driver is the real single point of failure (Chapter 2): the lineage and coordination live there.
The diagram traces a small lineage: a source RDD → filter → map. If Partition 1 is lost, Spark recomputes just that partition by replaying filter and map on its source slice.
The cost of recomputation is not always equal. When a lost partition sits behind a narrow dependency (each output partition depends on only one input partition), Spark recomputes just that one partition in isolation — cheap. When the lost partition sits behind a wide dependency (it was produced by a shuffle, where many input partitions contributed to one output partition), Spark may need to recompute entire upstream partitions and re-shuffle the data — much more expensive. The full definition of narrow vs wide belongs to Chapter 5; the point here is that lineage-based recovery has a cost structure, and you will feel it when jobs have long or wide-heavy lineages.
There is one limit to what lineage can recover: if the source data itself is gone, the recipe cannot be cooked. A local temp file deleted after the job started, a Kafka topic that has expired, or S3 data overwritten mid-job all break the assumption that the source is still readable. In those cases, checkpointing to reliable storage — saving a materialized snapshot of an intermediate RDD — gives Spark a recovery point that does not depend on replaying from the original source. See the caching-persistence-and-checkpointing chapter for the full treatment.
Core mental model
Lineage is a recipe, not a photograph. Lose a dish (partition) and you cook it again from the recipe, rather than restoring a saved copy. But if the ingredient (source data) has expired, even the recipe cannot save you — that is what checkpointing to reliable storage is for.
Why it matters
"How does Spark recover from node failure?" is a classic interview question, and the answer, recompute from lineage, also explains caching and checkpointing later (the caching-persistence-and-checkpointing chapter): they exist to shorten lineage that has grown too long, or to guard against sources that may disappear.
lineage (DAG)
The graph of transformations that produced an RDD; the basis for recomputation.
fault tolerance
Surviving failures; Spark recomputes lost partitions from lineage rather than restoring replicated copies.
recompute
Re-running the transformations for a lost partition instead of restoring a copy.
narrow dependency
Each output partition depends on exactly one input partition; a lost partition can be recomputed cheaply in isolation. (Full definition: Chapter 5.)
wide dependency
An output partition depends on multiple input partitions (produced by a shuffle); recovery may force recomputing entire upstream partitions. (Full definition: Chapter 5.)
checkpointing
Saving a materialized RDD snapshot to reliable storage (HDFS/S3) so recovery does not require replaying from the original source.
SOURCE RDD
raw partitions
from parallelize / read
→↓filter
RDD #2
filtered
keep matching rows
→↓map
RDD #3
mapped
transform each row
Lose Partition 1? Spark replays filter → map on just that source slice — the other partitions are untouched.
Each RDD records how it was built. If a partition is lost, Spark replays that lineage for just the missing partition on another executor — recovery without backups.
Common mistake
Thinking Spark replicates data for fault tolerance. It recomputes from lineage instead; the storage layer (HDFS/S3) handles replication, not the RDD.
Assuming lineage can always recover any lost partition. If the source data is ephemeral (a local temp file, an expired Kafka topic, overwritten S3 data), lineage cannot recompute the partition because the ingredient itself is gone. Checkpointing to reliable storage is the fix.
Better habit
Explain recovery as "recompute from lineage", not "restore from backup".
Watch for very long or wide-heavy lineages; recomputing a partition behind a shuffle forces upstream re-work.
Reach for checkpointing (caching-persistence-and-checkpointing chapter) when lineage grows unwieldy or when sources are ephemeral.
Interview note
"How is an RDD fault tolerant?" Answer: "It records its lineage, the transformations that built it, so if a partition is lost, Spark recomputes just that partition on another node from the lineage. The cost depends on the dependency type: narrow dependencies are cheap to recompute; wide dependencies (shuffles) can force upstream re-work."
Ephemeral sources break lineage recovery
If Spark built an RDD from a local file that was later deleted, a Kafka offset that has expired, or a temp path overwritten mid-job, lineage cannot recompute the lost partition — the source ingredient no longer exists. Checkpoint intermediate RDDs to S3 or HDFS whenever your source data could disappear before the job finishes.
Remember this
RDDs are resilient because they remember their lineage: lost partitions are recomputed by replaying transformations, not restored from backups. Recovery behind narrow dependencies is cheap; behind wide dependencies (shuffles) it is expensive. If the source itself is gone, lineage cannot help — checkpointing to reliable storage is the safety net.
Practice3 prompts
Walk through what Spark does when one partition is lost mid-job.
Why is recomputing a partition behind a wide dependency (shuffle) more expensive than one behind a narrow dependency?
In what scenario does lineage fail to recover a lost partition, and what is the fix?
04 · A taste of the API
RDD Transformations & Actions
Transformations write down steps; the action presses "run". Until the action, you have only a plan (lineage), not results.
⏱ 5 min · Topic 4 of 6
Even though you will mostly use DataFrames, a quick taste of the RDD API cements the transformation/action model. Transformations (map, filter, flatMap) are lazy, they build new RDDs and record lineage. Actions (collect, count, take, reduce) trigger execution and return a value to the driver.
The example creates an RDD from a list, filters to the even numbers, doubles them, and collects the result. Nothing actually runs until collect() is called, that single action triggers the whole chain across the partitions.
Because collect() pulls everything to the driver, it is only safe for small results, exactly the Chapter 2 warning. On big data you would write() from the executors instead.
One nuance worth noting: the transformation / action split is about when work happens, not how much. Some actions internally require significant data movement before they return — for example, sortBy must sort across all partitions before any result comes back. The full anatomy of narrow vs wide operations and how they map to stages is the subject of Chapter 5; we keep this section illustrative.
Core mental model
Transformations write down steps; the action presses "run". Until the action, you have only a plan (lineage), not results.
Why it matters
The transformation (lazy) vs action (triggers) distinction is the heart of how Spark runs, and Chapter 5 generalises it to DataFrames. Seeing it on tiny RDDs makes it unmistakable.
transformation
A lazy operation that returns a new RDD (map, filter, flatMap).
action
An operation that triggers execution and returns a value (collect, count, take, reduce).
collect()
Brings the entire RDD to the driver; safe only for small results.
take(n)
Returns the first n elements, a safe way to peek at large RDDs.
Keep evens {2, 4, 6}, double them → {4, 8, 12}. The filter and map only ran when collect() was called; collect() is safe here only because the result is tiny.
filter and map are transformations (lazy, lineage-only). collect() is the action that runs the chain across all partitions and returns the result to the driver. On large data, replace collect() with a write() so results stay distributed.
Common mistake
Expecting filter/map to run immediately. They are lazy; only the action (collect/count) triggers work, surprising if you expect eager execution.
Using collect() on a large RDD. It floods the driver and crashes it; use take(n) to peek or write() to persist.
Better habit
Build with transformations, trigger with a single action.
Use take(n) to peek; reserve collect() for genuinely small results.
On big data, end with write(), not collect().
Lazy by design
Laziness lets Spark see the whole chain before running it, so it can optimise (fuse narrow steps, prune work). You will see this pay off massively for DataFrames via Catalyst.
Remember this
RDD transformations (map/filter) are lazy and build lineage; actions (collect/count/take) trigger execution. collect() is for small results only; prefer take(n) or write().
Practice2 prompts
Which line in the example triggers execution, and what runs at that moment?
Replace collect() with a safe way to inspect a huge RDD.
05 · Choosing
RDDs vs DataFrames: When to Use Which
RDD = a bag of objects Spark cannot see inside. DataFrame = a table Spark understands and can optimise. Prefer the one Spark can optimise.
⏱ 4 min · Topic 5 of 6
If RDDs are the foundation, why do we mostly use DataFrames? Because DataFrames know the shape of your data (a schema) and run through the Catalyst optimizer, which rewrites your query into an efficient plan. RDDs are opaque to Spark, it sees lambdas, not columns, so it cannot optimise them the same way.
The practical rule: use DataFrames (and Spark SQL) for essentially all structured data work, joins, aggregations, ETL, analytics. Drop to RDDs only for the rare cases that need fine-grained control over partitioning or custom, unstructured processing the DataFrame API cannot express.
The comparison below summarises the trade-off. The headline: DataFrames are faster and shorter for almost everything, precisely because Spark understands them.
Core mental model
RDD = a bag of objects Spark cannot see inside. DataFrame = a table Spark understands and can optimise. Prefer the one Spark can optimise.
Why it matters
Choosing the right abstraction is a real performance decision: the same logic in DataFrames is often dramatically faster than in RDDs because Catalyst (Chapter 13) can optimise it. Knowing when each fits is a senior signal.
Catalyst optimizer
The engine that rewrites DataFrame/SQL queries into efficient plans (Chapter 13).
schema
The named, typed columns of a DataFrame; what lets Spark optimise.
Tungsten
Spark's execution layer that makes DataFrames memory- and CPU-efficient (Chapter 21).
declarative API
You describe what you want (select, groupBy); Spark decides how, the DataFrame style.
RDDs vs DataFrames
Aspect
RDD
DataFrame
Spark sees
Opaque functions (lambdas)
Columns + schema
Optimizer
None (you optimise by hand)
Catalyst rewrites your query
Code
More, lower-level
Less, declarative
Speed
Slower for structured work
Often faster (Catalyst + Tungsten)
Use when
Custom/unstructured, fine control
Almost everything else
Common mistake
Writing structured logic in RDDs out of habit. You forfeit Catalyst and Tungsten optimizations; the same job in DataFrames is usually much faster.
Better habit
Default to DataFrames / Spark SQL for structured data.
Drop to RDDs only when you truly need low-level control.
Remember DataFrames still run on RDDs, you lose nothing by going higher level.
Interview note
"RDD or DataFrame?" Answer: "DataFrame for almost everything, it has a schema and goes through Catalyst, so Spark optimises it. RDDs only for custom, low-level work the DataFrame API can't express."
Remember this
Prefer DataFrames: their schema lets Catalyst and Tungsten optimise them, so they are faster and shorter. Use RDDs only for rare low-level or unstructured needs, knowing DataFrames run on them anyway.
Practice2 prompts
Why can Spark optimise a DataFrame query but not an equivalent RDD one?
Name a situation where dropping to RDDs is actually justified.
06 · Next Chapter
Next Chapter
You have met partitions, lineage, and the transformation/action split on RDDs, the foundation under every DataFrame.
⏱ 3 min · Topic 6 of 6
Next chapter
Transformations, Actions & Lazy Evaluation
You have met partitions, lineage, and the transformation/action split on RDDs, the foundation under every DataFrame.
Next, Chapter 5 makes lazy evaluation precise: narrow vs wide transformations, how the lazy DAG is built, and how an action turns it into the stages and tasks you saw in the Spark UI.
