What You'll Master Here
An RDD is a single logical collection cut into partitions, each living on a worker. It remembers how it was built (its lineage), so any lost partition can be recomputed instead of restored from a backup.
Underneath every DataFrame, every Spark SQL query, and every streaming job sits one foundation: the RDD, the Resilient Distributed Dataset. You will rarely write RDD code today, but understanding RDDs is what makes partitions, lineage, and fault tolerance click, and those concepts power everything above them.
This chapter explains what an RDD actually is (a partitioned, immutable collection spread across the cluster), how Spark recovers from machine failure for free using lineage, and why we almost always prefer DataFrames now, while still standing on RDDs underneath.
We keep it concrete with a tiny RDD example you can run, plus diagrams of how an RDD is partitioned and how a lost partition is recomputed.
An RDD is a single logical collection cut into partitions, each living on a worker. It remembers how it was built (its lineage), so any lost partition can be recomputed instead of restored from a backup.
Partitions, lineage, and fault tolerance are RDD ideas that resurface in every later chapter and many interviews ("how does Spark recover from a failed node?"). Learn them here once and the rest of Spark rests on solid ground.
- RDD
- Resilient Distributed Dataset: an immutable, partitioned collection processed in parallel.
- partition
- One slice of an RDD; each is processed by one task on one core.
- lineage
- The recorded chain of transformations that built an RDD; used to recompute lost data.
- immutable
- RDDs never change in place; a transformation creates a new RDD.
Reaching for RDDs for everyday data work. You lose Catalyst optimization and write more code; prefer DataFrames unless you truly need low-level control.
Think of any dataset as "partitions across the cluster", that is the RDD view.
Explain fault tolerance via lineage (recompute), not backups.
Default to DataFrames; treat RDDs as the foundation, not the daily tool.
An RDD = partitions + lineage. Partitions give parallelism; lineage gives fault tolerance. DataFrames add a schema and an optimizer on top, but inherit both.
An RDD is an immutable, partitioned, fault-tolerant collection. Partitions enable parallelism; lineage enables recompute-on-failure. DataFrames are built on this foundation.
- Define an RDD in one sentence using "partitions" and "lineage".
- Why do we prefer DataFrames day to day if RDDs are the foundation?