What You'll Master Here
Catalyst plans with a map drawn from estimates. AQE is a driver that re-checks the map at each junction using what it now actually sees, and reroutes. Plan, observe, adapt.
Catalyst (Chapter 13) optimises a query before it runs, using estimates. But estimates can be wrong, real data sizes only become known at runtime. Adaptive Query Execution (AQE) fixes this: it re-optimises the plan during execution using actual statistics gathered after each shuffle. It is the most impactful Spark performance feature of the last several years, and it is on by default in Spark 3.2+.
This chapter covers AQE's three big tricks, dynamically coalescing shuffle partitions (fixing the wrong-200-default problem automatically), switching a sort-merge join to a broadcast join when a side turns out small, and splitting skewed partitions, plus cost-based optimisation (CBO), which uses collected table statistics to make better plan choices (like join ordering).
With a diagram of static vs adaptive planning and examples of AQE in action. The headline lesson: enable AQE, it solves several earlier-chapter problems for free.
Catalyst plans with a map drawn from estimates. AQE is a driver that re-checks the map at each junction using what it now actually sees, and reroutes. Plan, observe, adapt.
AQE automatically handles partition sizing (Chapter 16), skew (Chapter 18), and join strategy (Chapter 20), problems you would otherwise tune by hand. Knowing what it does (and that it is on by default) is essential modern Spark knowledge.
- AQE
- Adaptive Query Execution: re-optimising the plan at runtime with real stats.
- partition coalescing
- AQE merging small shuffle partitions into right-sized ones automatically.
- dynamic join switch
- AQE changing sort-merge to broadcast when a side turns out small.
- CBO
- Cost-Based Optimization: using table statistics to choose better plans.
Manually tuning shuffle partitions and skew while AQE is disabled. You do by hand what AQE does automatically, enable it first, then tune only what remains.
Keep AQE enabled (default on in Spark 3.2+).
Let AQE coalesce shuffle partitions instead of hand-setting 200.
Collect table statistics (ANALYZE TABLE) so CBO can help.
Static optimisation guesses; AQE measures. By re-planning at runtime with real shuffle statistics, it auto-fixes partition sizing, skew, and join strategy, three things earlier chapters tuned manually.
AQE re-optimises the plan at runtime using real statistics, automatically coalescing shuffle partitions, switching join strategies, and handling skew. Combined with CBO's statistics-driven choices, it resolves several tuning problems for free, so enable it.
- Why can runtime re-optimisation beat static (plan-time) optimisation?
- What three problems does AQE address automatically?