How D8Loop works: Spark Playground

A five-minute guided tour · Spark Playground

Run it. See what Spark did. Explain it.

Most Spark prep asks you to recite what a shuffle is. The Spark Playground gives you your own Spark and a notebook, and shows you the jobs, the plan and the tasks for every cell you run. This tour replays one real scenario, Two counts, two scans, recorded on the playground.

About twenty clicks, and most of them are a call about what Spark will do. Follow the pointer — or use Back and Next in the guide.

Read the tour as text

How D8Loop works — Run it. See what Spark did. Explain it.. Most Spark prep asks you to recite what a shuffle is. The Spark Playground gives you your own Spark and a notebook, and shows you what Spark did with each cell you run. This tour replays one real scenario, recorded on the playground, on the screens you will use.
Step 1 · Pick a scenario — 111 scenarios, in three levels. The catalogue groups 111 scenarios into Foundations, Intermediate and Advanced, in learning order. Each one is an interview question you answer by running PySpark and reading what Spark did. This tour takes the first: two counts, two scans.
Step 2 · Start a session — A private machine, Spark and a notebook. Starting a session boots a private machine, starts Spark and JupyterLab, and opens the scenario’s notebook. It takes about forty seconds, so Sparky keeps you company, and the checklist follows the real start-up, never a timer.
Step 3 · Read the brief — The notebook opens on the brief. The first cell of the notebook is the brief: a situation, a task, the dataset, and the interviewer’s question. The phrase to spot is the belief being tested — that after one count the data is already loaded.
Step 4 · Run the transformations — Three cells, and Spark barely moves. Run the read, the filter and the select, one cell at a time. The panel on the right follows the last cell you ran. The read runs one small job to learn the schema; the filter and the select run none — What happened calls it lazy evaluation.
Step 5 · Run the action — count() is where the work happens. count() is an action, so Spark finally runs the plan: two jobs, three stages with one skipped, and three tasks, over a million rows of events. What happened explains each piece in Spark’s own words, with the rows out of every operator.
Step 6 · Job DAG — Two jobs, three stages, one of them skipped. The Job DAG tab shows the jobs the cell started and their stages. With adaptive query execution on, the stage before the shuffle runs as its own job; the second job lists it again, skipped, and runs only the final stage.
Step 7 · SQL plan — Spark’s own operators, in Spark’s own words. The SQL plan tab is the physical plan Spark ran: Scan parquet, ColumnarToRow, Filter, Project and a partial HashAggregate fused into WholeStageCodegen (1), an Exchange that ends the stage, then the final HashAggregate. The names are the ones in explain() and in interviews.
Step 8 · Stages & tasks — Run it again: did the second count reuse the first?. Stages & tasks gives every stage its task times — min, median and max — and the data it read. Run the same count again and compare: the second is quicker, and it still reads every row from the files. Time is not the evidence; the rows read are. A task far slower than the median would be flagged as skew.
Step 9 · Answer and mark it done — Four questions, an explained solution, and your plans. The notebook ends with the scenario’s questions and where to look for each. The explained answers are in solution.ipynb next to it. Mark the scenario done once Spark has run in your session, and it counts in your dashboard plans.
That is the loop — Run. Read. Explain. Then the next scenario.. Everything in the tour is a recording of the real product: the scenario’s own notebook, and the jobs, stages, plan and task times Spark reported for it, explained by the same What happened panel. In a session the notebook is yours to change, and the panel follows whatever you run.