What You'll Master Here
groupBy gathers rows of the same key together (a shuffle); agg then computes one summary row per key. Spark pre-aggregates per partition first to move less data.
Aggregation is where raw rows become metrics: counts, sums, averages, per group. It is the heart of analytics and one of the most-tested skills in interviews. This chapter covers groupBy + agg, running several aggregations at once, rollup and cube for subtotals, and pivot for turning rows into columns.
Aggregations are also your first heavy, recurring shuffle: groupBy redistributes rows so each group lands together (Chapter 5), then computes. Spark softens this with partial aggregation (it pre-aggregates within each partition before the shuffle), which you will see in the plan.
Every pattern comes with input and output so the transformation is unmistakable.
groupBy gathers rows of the same key together (a shuffle); agg then computes one summary row per key. Spark pre-aggregates per partition first to move less data.
Grouped aggregation is the backbone of reporting, metrics, and analytics, and a staple of SQL/Spark interviews. Doing it correctly and efficiently is core data-engineering competence.
- groupBy
- Groups rows by one or more key columns; a wide (shuffle) transformation.
- agg
- Computes one or more aggregate expressions per group (count, sum, avg…).
- rollup / cube
- Aggregations that also produce subtotals (rollup) and all combinations (cube).
- pivot
- Turns distinct values of a column into separate columns.
Aggregating before filtering/cleaning. You shuffle and summarise junk (un-normalised keys, nulls), producing wrong metrics. Clean first (Chapters 8–9).
Filter and normalise keys before grouping.
Name every aggregate with .alias() for clean output columns.
Expect groupBy to be a shuffle; minimise the data it sees.
groupBy + agg = rows → metrics. It is a shuffle, so feed it the least, cleanest data possible. rollup/cube add subtotals; pivot reshapes rows into columns.
Grouped aggregation (groupBy + agg) turns rows into per-group metrics via a shuffle. rollup/cube add subtotals; pivot reshapes. Clean and shrink data before aggregating.
- Is groupBy narrow or wide, and what does that imply for performance?
- Why clean/normalise keys before aggregating?