What Apache Spark Is & Why It Exists

Apache Spark is a tool that runs one data job on many computers at the same time. You write the job once. Spark splits the work.

You already know SQL, Python, batch and streaming, storage, and pipeline design. Spark is the engine that can run those steps when the data is too big for one computer.

This chapter shows the main idea. You will do one job, revenue by customer, in plain Python, in SQL, and in Spark. Then you will see why Spark splits data, and when you should not use it.

Later chapters this week explain each part.

• The architecture chapter shows who does the work.
• The lazy evaluation chapter shows why Spark waits before it runs.
• The DataFrames chapter shows DataFrames in more depth.
• The shuffle chapter shows the costly step called the shuffle.

A machine is one computer. It has a fixed amount of memory (RAM), disk space, and processor time. Every tool you used so far runs on one machine.

This is fine for the shop. The real shop gets about 200 orders a day. That fits easily in Postgres or in pandas (a Python table library) on a laptop.

Now make an assumption. Imagine the shop had hundreds of millions of orders a day, kept for years.

This is not true for our shop. It is only a way to see the problem.

With that much data, three things go wrong on one machine.

1. The data does not fit in memory, so the job stops with an error or runs very slowly.
2. The data does not fit on one disk.
3. One machine must read every row by itself, so the job takes too long.

A bigger machine helps for a while. At some size it becomes very costly, and then it stops being possible. The other way is to use many normal machines together.

A group of machines that work together is called a cluster. The hard part is sharing the work between them. Spark does that part for you.

The job is: total paid revenue for each customer. Here is the input. We use it in all three tools.

orders = [
    (101, 1, "paid", 40.00, "2026-01-03"),
    (102, 1, "paid", 25.00, "2026-01-10"),
    (103, 2, "refunded", 60.00, "2026-01-11"),
    (104, 3, "paid", 15.50, "2026-01-15"),
    (105, 3, "pending", None, "2026-01-20"),
    (106, 4, "paid", 80.00, "2026-01-21"),
]

revenue = {}
for order_id, customer_id, status, amount, order_date in orders:
    if status == "paid":
        revenue[customer_id] = revenue.get(customer_id, 0) + amount

for customer_id in sorted(revenue):
    print(customer_id, f"{revenue[customer_id]:.2f}")

Now the same job in SQL. You know this one. It says what you want, not how to loop.

SELECT customer_id,
       SUM(amount) AS revenue
FROM   orders
WHERE  status = 'paid'
GROUP  BY customer_id
ORDER  BY customer_id;

PySpark lets you write Spark code in Python. A SparkSession is your starting point in Spark. A DataFrame is a table that Spark can spread over many machines.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("shop").getOrCreate()

orders = spark.read.csv("orders.csv", header=True, inferSchema=True)

revenue = (
    orders
    .filter(F.col("status") == "paid")
    .groupBy("customer_id")
    .agg(F.sum("amount").alias("revenue"))
    .orderBy("customer_id")
)

revenue.show()

• F is the functions module, imported as F. It gives you col and sum.
• header=True: the first line has the column names.
• inferSchema=True: Spark guesses each column's type.
• filter = WHERE. groupBy = GROUP BY. agg(F.sum(...)) = SUM. alias = AS. orderBy = ORDER BY.
• show() prints the result.

The answer is the same in all three tools. The difference is where the work can run.

Python and SQL ran on one machine. Spark can run the same steps on a whole cluster.

Spark does not filter or add up the rows when it reaches those lines. It waits until show(). The lazy evaluation chapter explains why.

Spark splits data into pieces called partitions.

This is not the same as the date folders from the storage chapter. Those folders are on disk. A Spark partition is a group of rows that one machine works on during a job.

Each machine works on some partitions at the same time. Then Spark combines the partial answers. A partial answer is the total from one partition only.

Our shop is tiny, so we use two partitions of three rows to see the idea. Machine A gets orders 101, 103, 105. Machine B gets orders 102, 104, 106.

Each machine runs the same steps on its own rows only.

Look at customer 1. Machine A sees order 101. Machine B sees order 102.

def partial_revenue(rows):
    totals = {}
    for order_id, customer_id, status, amount in rows:
        if status == "paid":
            totals[customer_id] = totals.get(customer_id, 0) + amount
    return totals

machine_a = [(101, 1, "paid", 40.00), (103, 2, "refunded", 60.00), (105, 3, "pending", None)]
machine_b = [(102, 1, "paid", 25.00), (104, 3, "paid", 15.50), (106, 4, "paid", 80.00)]

part_a = partial_revenue(machine_a)   # {1: 40.0}
part_b = partial_revenue(machine_b)   # {1: 25.0, 3: 15.5, 4: 80.0}

final = {}
for part in (part_a, part_b):
    for customer_id, total in part.items():
        final[customer_id] = final.get(customer_id, 0) + total

This is a model, not Spark code. Real Spark does the splitting, the running, and the combining for you. You never write the two machines yourself.

The combine step is the costly part.

The partial totals for the same customer must meet on one machine. Spark sends them over the network. The shuffle chapter teaches this step.

Spark is one engine with several parts on top. You use the same engine for different kinds of work.

Spark also keeps data in memory between steps when it can.

An older tool, Hadoop MapReduce, also ran jobs on many machines. It wrote results to disk after each step. That made jobs with many steps slower.

Spark reads Parquet files and the date folders you saw in the storage chapter.

Often the answer is no. Spark adds work: you set up a cluster, wait for it to start, and learn how it runs.

Our shop has about 200 orders a day. That fits in Postgres or pandas on one machine. Spark would be slower to start and harder to run, with no gain.

You can run Spark on one machine too.

This is called local mode. Spark still splits the work, but one machine does all of it. It is good for learning and testing.

Check yourself with these questions.

1. In the two-machine example, why did customer 1 need a combine step?
2. Is Spark a database? Say why or why not.
3. Name one job you would run on Spark and one you would not.