Loadingone moment…
Apache Spark · Playground
Scenario questions you answer in a real notebook, reading the DAG, plan, stages and tasks Spark produced for your code.
Your own notebook and Spark, with datasets shaped to show skew, shuffles and joins.
The job DAG, SQL plan and stages Spark drew itself, for the cell you just ran.
Min, median and max per stage, so skew and shuffle cost stop being abstract.
Interview questions per scenario, with explained solutions you can run.
Lazy evaluation, shuffles, joins and AQE, explained first: 30 chapters that make every DAG below readable.
Open the knowledge baseA teammate counts purchase events twice in one notebook and is surprised the second count is no faster than the first.
Two queries over the same customers table: one keeps gold customers, the other lists the distinct countries.
The orders table is eight Parquet files, so a colleague expects every scan of it to run eight tasks.
A daily job reads a CSV export with inferSchema=True, and its first "read" line shows jobs in the Spark UI before any action.
A code review comment says "always select only the columns you need before an aggregation, it makes Spark read less".
Someone on the team believes Parquet can only skip data for equality filters, so they avoid range filters on amount.
A report of "customers outside the US" is short by every customer whose country was never filled in.
You need one customer row per country, and a colleague says dropDuplicates(["country"]) is "just distinct on one column".
Writing all orders sorted by amount shows three jobs in the Spark UI, and the first one finishes before the "real" work starts.
Revenue per country was one groupBy; now finance wants it per country and status.
A dashboard shows unique customers per day, and the query gets slower as the business grows.
A metrics job computes sum, average, count, max and min of order amount per country.
Finance wants delivered revenue per country, highest first.
Joining every order to its customer runs faster than a teammate expected, and nobody wrote broadcast() anywhere.
A colleague changes an inner join to a left join "to be safe" and a right join "for symmetry", and the right join is suddenly much slower.
You need orders placed from the customer's home country, so the join must match customer_id and country.
A wishlist report built with explode() is missing every customer whose wishlist is empty or missing, and nobody noticed because nothing failed.
Your team stores customer attributes as a nested profile struct and an attrs map, and a reviewer worries that reading one field now reads the whole blob.
A monthly revenue job derives the month, year, date, day number and weekday from order_ts.
Customer names look like customer_123, and you need the number, the prefix and an uppercase label.
An order-labelling job has two when/otherwise chains, and someone proposes splitting them into separate DataFrames "so Spark does less per step".
A teammate filters orders down to the delivered ones, runs two queries on the result, and adds .cache() "to make it faster".
Your job cached a filtered DataFrame, used it twice, and then called unpersist() to free memory.
A downstream team wants the customers table as a single Parquet file, and another wants six evenly sized files.
A colleague insists the DataFrame API is faster than writing Spark SQL strings, so every query should be rewritten.
An analyst registers a filtered DataFrame as a temp view so the rest of the team can query it in SQL, and assumes the view holds the filtered rows.
You want a quick look at five orders before writing a real query.
A data scientist builds a 1% sample of orders for a model, reruns the notebook the next morning, and gets different rows, even after "adding a seed".
Sales wants the three biggest orders in every country, and the classic answer is row_number() over a window partitioned by country.
Every Spark tuning conversation ends with "show me the explain output", but explain() can print five different things.
A generated query is full of leftovers like WHERE 1 + 1 = 2 and multiplications by literal constants, and a reviewer worries Spark evaluates them for every row.
A nightly job that writes the products table failed halfway, and someone re-ran it.
Most queries on orders ask about one country, so the team rewrites the table partitioned by country.
An upstream job wrote orders as dozens of tiny files and the jobs reading it got slower, although the data did not grow.
Two dashboards report different numbers of signups for the same day: one uses count("*"), the other count("referral_tier").
You compute the average price per product category and then filter the result twice: once on the average, once on the category name.
You pivot order revenue so each status becomes a column, and the Spark UI shows jobs running on a line that has no action.
A feature pipeline builds each customer's purchase history with collect_list and assumes it is in time order, because the orders were sorted by order_ts first.
A partner delivers orders as JSON files, and your job filters them exactly as it filters the Parquet copy.
"Parquet is faster than CSV" is the most common file-format answer in interviews, and the follow-up is always "why, exactly?".
Finance wants a running revenue total per country over time, and the obvious sum().over(Window.partitionBy().orderBy()) looks right on the first rows.
A join between two large inputs is spending most of its time sorting, and a teammate suggests adding a SHUFFLE_HASH hint.
Your platform team disabled automatic broadcast joins cluster-wide after one blew up a driver, and now a small lookup join shuffles 500k orders.
Revenue by customer tier and product category needs the orders fact table joined to two dimensions.
Support wants two lists: customers who have cancelled at least one order, and customers who never have.
Marketing wants every pair of gold customers who live in the same country, for a referral campaign.
An analyst wrote "customers with at least one order over 800" as a correlated EXISTS subquery and worries it runs the inner query once per customer.
Two monthly extracts were written by different people, one with columns (order_id, country, status) and one with (order_id, status, country).
You need gold customers who have cancelled an order, and gold customers who never have, and set operations read naturally: gold.intersect(cancelled) and gold.subtract(cancelled).
Product wants the three cheapest orders in each country.
Growth wants to know how often users come back after a long silence, which means computing the time since each user's previous event.
A dashboard shows a "rolling 3-order average" of order value, written as avg("amount").over(Window.partitionBy("country").orderBy("order_ts")).
A row_number per country is fast in tests and slow in production, and the Spark UI shows one task running far longer than the rest.
A feature pipeline adds window columns one withColumn at a time: a rank per country, a rank per status, then a running total per country.
A notebook computes customer spend from delivered orders, then joins it to customers twice: average spend by tier and by country.
An iterative job keeps building on the previous step's DataFrame, and its plans grow until planning itself gets slow.
A data-quality job counts bad rows with an accumulator while it filters them out, and the dashboard shows twice as many bad rows as the source has.
A teammate says "I already broadcast the country lookup" and points at sc.broadcast inside a UDF, while the code review asks for a broadcast join.
A teammate wrote the sales-tax column as a Python UDF because "it is just one line".
The team has to keep some logic in Python and is told to "use a pandas UDF, it is vectorized".
A pipeline calls repartition(8, "country") right before groupBy("country"), with a comment saying it "pre-partitions so the aggregation does not shuffle again".
Revenue per customer tier needs orders joined to customers and summed.
A report unions large and small orders, and each branch joins the same customers table.
A "latest order per customer" table is built with dropDuplicates(["customer_id"]), and it passed every test.
Orders are heavily concentrated in India, and the team blames "skew" for every slow job that touches country.
In the events table, user 0 (a bot account) produces about 30% of all rows.
An engagement score per user is computed with a pandas aggregate UDF, and the stage always waits on one task: the bot account user 0 owns most of the rows.
The same delivered-revenue report runs against three copies of the orders data: Parquet, ORC and JSON.
In a debugging interview you are handed the explain("formatted") output of a join-and-aggregate query and asked to walk through it: what each numbered node does, which columns flow where, and what changed once the query actually ran.
Every tuning guide says to set spark.sql.shuffle.partitions, and every Spark 3+ guide says AQE fixes partition counts for you.
A report keeps every customer and attaches their orders with a left join.
You paste explain() output into a performance review, and a colleague points out it does not match the SQL tab of the Spark UI.
A join between all orders and one hour of events is planned as a sort-merge join, even though that hour holds only a few dozen events.
A small aggregate over one country shuffles into 8 partitions but its final stage runs a single task.
Thirty percent of all events belong to user 0, so a join of orders and events on the user puts a huge hash partition in one task.
Most orders come from India, so a shuffle join on country sends a huge share of the rows to one task.
Sales data is partitioned by country, and a report joins it to a small region table filtered to EMEA.
Two large tables are joined on customer_id by dozens of jobs a day, and every one of them shuffles both sides.
An enrichment job adds customer tier and product category to every order and writes the result, with no aggregation.
An interviewer asks what the *(1) and *(2) in a physical plan mean, and then hands you a plan where one operator has no star at all.
A pipeline chains filter() calls in whatever order people added them, filters on derived columns, and filters after joins.
A report joins orders to customers and products and keeps just two columns at the end.
A job that sorts a few million rows got slower after the data grew, with no errors.
Shopping baskets arrive as an array of item structs plus an array of tags per customer.
Each order carries an array of amounts, and the team wrote a Python UDF to apply tax to every element.
Product wants sessions: a new session starts when a user has been idle for more than 30 minutes.
You already know that rank() <= 3 lets Spark keep only the top rows of each country before the shuffle.
A dashboard needs the median order amount per country every hour.
A report ranks orders within each country and within each country and status.
An analyst filters customers with a correlated scalar subquery, (SELECT COUNT(*) FROM orders WHERE ...) > 12, and a reviewer fears it runs once per customer.
A three-table query joins orders to customers and products with a filter on each dimension.
A customer report puts each customer's total spend next to their cancelled spend, so the query reads orders twice.
A scoring job updates every customer's score in a Python loop, each step blending in the average score of their country.
After a source system change, a quarter of the orders arrive with no customer_id.
An upstream team added a discount column to new order batches without telling anyone.
Storage for the orders lake has doubled in a year, and someone proposes switching every Parquet write from snappy to gzip.
A notebook that worked for months started failing after the data grew, on a line that just says collect().
A feature pipeline builds, per customer, the list of products they bought and the set of countries they ordered from.
The interview question: "for every order, show its amount as a percentage of its country's total".
A daily job filters delivered orders, joins customers, and feeds three reports: revenue by tier, by home country and orders by month.
A view ranks every order within its country by amount, and analysts filter that view afterwards: some by country, some by status.
A join was slow, so someone wrapped the bigger side in F.broadcast() and it got faster.
A reviewer approves a query because explain() shows PushedFilters, "so Parquet skips the data".
Three pipelines compute total spend per customer and one of them is "slow for no reason".
A dashboard shows revenue per country for each order status.
Finance labels every order with a size band from a small table of lower and upper bounds, joined on amount between them.
HR wants every employee who reports to manager 2, directly or through any number of levels.
A data scientist computes, per country, the share of revenue that comes from the ten largest orders, with groupBy().applyInPandas because it is easy in pandas.
A team upgraded from Spark 2 and wants to know what adaptive query execution actually does for their jobs before they tune anything.
A take-home review: a colleague's pipeline computes gross delivered revenue for gold customers three ways, and it is far slower than it should be for this data.
The final interview round: "Here is a real job: the three largest delivered orders from gold customers in each country, with product category, top ten overall.