What You'll Master Here
pick columns (select/selectExpr), keep rows (filter), compute new columns (withColumn/lit/cast/when), then maybe order/trim. Do the cheap, shrinking steps first, and treat null as a value that needs a deliberate decision at every step.
This is the bread and butter of Spark: the handful of DataFrame operations you will use in literally every job. select, selectExpr, and filter to choose columns and rows; withColumn, lit, cast, when/otherwise, withColumnRenamed, and drop to shape columns; and sort, distinct, and limit to order and trim.
None of this is hard, but doing it cleanly, knowing which operations are cheap (narrow) versus expensive (wide), and knowing where nulls quietly change your answer, is what separates readable, fast, correct code from slow, tangled, silently-wrong code. Recall from Chapter 5 that wide transformations trigger a shuffle while narrow ones do not — that distinction is exactly what tells you which of these everyday operations to reach for freely and which to use sparingly. We will also cover chaining operations into a clear pipeline.
Every operation comes with input and output so you can see exactly what it does, including the traps: a null that disappears from a filter, a cast that returns null instead of erroring, a dropDuplicates that keeps a different row each run.
Most jobs are: pick columns (select/selectExpr), keep rows (filter), compute new columns (withColumn/lit/cast/when), then maybe order/trim. Do the cheap, shrinking steps first, and treat null as a value that needs a deliberate decision at every step.
These operations make up the large majority of day-to-day Spark transformations. Writing them clearly, ordering them well (filter early, sort late), and knowing their edge cases is what makes your jobs readable, efficient, and trustworthy.
- select / selectExpr
- Choose / compute the columns to keep, with typed Columns or SQL-expression strings.
- filter / where
- Keep only the rows matching a condition (identical methods).
- withColumn
- Add or replace one column from an expression.
- orderBy / sort
- Sort rows; a wide transformation (it shuffles).
Filtering and selecting late, after heavy steps. You carry more data than needed into joins/shuffles; filter and select early to shrink data first.
Filter and select as early as possible to reduce data.
Use withColumn for derived fields; select to project.
Remember orderBy and distinct are wide (shuffles), use them only when needed.
A few composable operations cover most transformations. The skill is ordering them well, shrink early with filter/select, avoid needless wide ops like sort and distinct, and handle nulls and casts deliberately rather than by accident.
select/selectExpr/filter/withColumn/cast/when/rename/drop/sort/distinct/limit cover most transformations. Apply the cheap, data-shrinking ones early; reserve wide ones (sort, distinct) for when you truly need them; and watch nulls at every step.
- Which of these are narrow and which are wide: select, filter, orderBy, withColumn, distinct?
- Why should filter usually come before a join?