Scaling & Performance

A pipeline that works for one day of orders can fail for one year of orders. This chapter is about the whole pipeline, not one tool.

To scale means to handle more data in the same time. A senior engineer works in a fixed order: measure, do less work, add machines, then check the cost.

You already know the tool details. Here is a one-line recap of each, so we do not teach them again:

• Spark: Partitioning & Parallelism (split data into pieces), The Shuffle: Spark's Performance Killer (move data), Data Skew & Salting (one piece too big).
• Spark: Performance Tuning & Debugging with the Spark UI, Adaptive Query Execution & Cost-Based Optimization, and Spark Configurations tune one job.
• Spark: Reading & Writing Data: Sources & File Formats, and Delta Lake & the Lakehouse (how files and tables are written).
• SQL: Warehouse Performance & Cost covers speed and cost inside a warehouse.
• Pipelines: Storage Layers & File Formats (where data lives), and Incremental Processing & Backfills (load new data, rebuild past days).
• Pipelines: Streaming Architecture Fundamentals (data that never stops arriving), and Data Observability & Monitoring (watch your pipelines as they run).

What you will learn

1. How to find the slow stage before you change anything.
2. Scale up or scale out.
3. Why parallel work has a limit.
4. How to split work by date or key.
5. How to load only what changed.
6. How to run a backfill without overloading a source.
7. Small files and compaction.
8. How to estimate the volume.
9. How speed costs money.

We use the shop. The six orders are real. Every large number is made up, and we say so each time.

A pipeline has stages. Extract reads data from a source.

Transform changes the data. Load writes it to the destination. A worker is a machine or process that does this work.

The bottleneck is the stage that limits the speed of the whole pipeline. Speeding up a smaller stage can save only a small part of the total.

So you measure first. Here is a nightly orders job, with the minutes each stage took (made up). The last row is time spent waiting for a free slot, where one task (a small piece of work) can run:

SELECT stage,
       minutes,
       100 * minutes / SUM(minutes) OVER () AS percent
FROM   stage_times
ORDER  BY minutes DESC, stage;

Extract is the bottleneck, so work on extract first. More workers for transform could save at most 18 of the 90 minutes.

Now ask which resource is slow inside that stage. Look at the five places a stage can be held up:

A cluster is a group of machines that work on one job together. Each machine has cores. A core runs one task at a time.

You have two ways to get more power. Scale up means a bigger machine.

Scale out means more machines. Look at how they differ:

A guideline: if the job fits on one machine and the stage is slow, try scale up first. It is simple. Scale out when one machine is no longer enough, or when the work splits well.

Remember the queue row from the last section. Tasks wait when there are more tasks than slots.

More slots shorten that wait. So does starting fewer jobs at the same time.

Parallel means many workers do parts of a job at the same time. Some parts cannot be shared. They run one after another, for example the final step that saves the result.

Say a transform takes 60 minutes. Six minutes cannot be shared. The other 54 minutes can be split between any number of workers.

This simple model gives the time with N workers: 6 + 54 ÷ N.

It is a model, not a measurement. Real jobs also pay to coordinate the workers. Try five sizes:

SELECT workers,
       ROUND(6 + 54.0 / workers, 2)        AS minutes,
       ROUND(60 / (6 + 54.0 / workers), 2) AS speedup
FROM   workers_to_try
ORDER  BY workers;

Going from 10 to 100 workers is 10 times the machines. The speedup grows only from 5.26 to 9.17. The six minutes that cannot be shared set a ceiling: 60 ÷ 6 = 10 times, even with endless workers.

So find the part that cannot be shared and make it smaller. Other limits act the same way: a source that allows few connections, or an API quota.

Spark teaches this at the engine level in Partitioning & Parallelism. Here you use it to decide how many workers a job deserves.

To run work in parallel, cut it into pieces that do not depend on each other. Two common cuts are by date and by key.

A key is the column you split on, for example customer_id. All rows with the same key go into the same piece. Here are the real shop orders, counted by customer:

SELECT customer_id,
       COUNT(*) AS orders
FROM   raw_orders
GROUP  BY customer_id
ORDER  BY customer_id;

Pieces are not always equal. Say the shop has 2,000,000 orders in a day (made up), split into four pieces. One customer, 230, is a large business:

Four workers start together, but the job ends when the worker with customer 230 ends: 240 seconds. An even split would give 500,000 rows to each worker, which is 100 seconds. This uneven split is called skew, and it takes 2.4 times longer.

Choose a key with many values and no giant value. A date is a good key for events. Watch for one huge day, such as a sale.

The cheapest work is the work you skip. A full reload reads all history every run. An incremental load reads only the new or changed rows (Incremental Processing & Backfills).

Here is the incremental job for one day of the shop. It asks only for 2026-01-15:

SELECT order_date,
       SUM(amount) AS revenue
FROM   raw_orders
WHERE  status = 'paid'
  AND  order_date = '2026-01-15'
GROUP  BY order_date;

Now see the size of the gain. Say the shop makes 2,000,000 orders a day for a year (made up).

One worker reads 5,000 rows a second (made up too). Compare the two loads:

The year has 365 days, so 365 × 2,000,000 = 730,000,000 rows. The full reload also grows every day. The incremental load stays near the size of one day.

A backfill runs the job again for past days. Say you must rebuild 90 days.

One day takes 400 seconds. One run at a time takes 90 × 400 = 36,000 seconds, which is 10 hours.

You can run several days at once. See how the time changes (made up: each run uses one source connection and runs at full speed):

Rounds are 90 ÷ runs. Total time is rounds × 400 seconds. But ten runs at once is only allowed if the source can take ten connections.

Three limits stop you from going faster:

• The source. A database can allow only a few connections. A backfill that uses them all slows the daily run and the people who use that database.
• An API quota. A rate limit is the number of calls allowed in a time window. Above it, the API refuses calls, often with the error code 429.
• The orchestrator, the tool that schedules your tasks. It can limit how many go at once. Airflow has pools and a limit on active runs.

Say an API allows 600 calls an hour (made up), and each day needs 20 calls. Then 90 days need 1,800 calls. That is 3 hours at best, however many runs you start.

When the API says 429, wait and try again with a longer wait each time. Keep the number of runs at once below the source's connection limit. Keep the daily run first in line.

Every writer makes its own files. More parallel writers and more frequent runs make more files. You met this in Storage Layers & File Formats and Delta Lake & the Lakehouse.

Say a job runs 24 times a day and each run writes 50 files (made up).

One day holds 200 MB of data (we work this out in the next section). Look at the two layouts:

A day has 24 × 50 = 1,200 files. A year has 1,200 × 365 = 438,000. Each file is 200 MB ÷ 1,200, which is about 0.17 MB.

Compaction means you rewrite many small files into a few larger ones. Each file costs a request to open and a line in the table metadata. So reading 438,000 files takes much more extra work than reading 730 files.

Run compaction as its own job, at a quiet time. Use your table's maintenance command, such as OPTIMIZE in Delta Lake. Aim for about 100 MB to 1 GB per file; your engine's guide gives its number.

Capacity planning means you estimate the data size before it becomes a problem. The formula is short: rows a day × bytes a row = bytes a day.

The six real shop orders are tiny, so we use made-up numbers. Say the shop makes 2,000,000 rows a day.

Say a row is 100 bytes (a guess; measure a real table). That is 200,000,000 bytes, or 200 MB, a day.

Over 365 days that is 200 × 365 = 73,000 MB, or 73 GB.

Now add growth. Say orders grow 5% each month (made up).

After 6 months the volume is 1.05 multiplied six times, about 1.34. After 12 months it is about 1.80.

Here is the plan, rounded. The seconds column uses the made-up speed of 5,000 rows a second for one worker:

SELECT month,
       rows_per_day,
       rows_per_day * 100 / 1000000 AS mb_per_day,
       rows_per_day / 5000          AS seconds
FROM   volume_plan
ORDER  BY month;

Now ask when it breaks. Say the job must finish in 60 minutes.

One worker reads 5,000 × 3,600 = 18,000,000 rows in that time. That is 9 times today.

At 5% growth a month, 9 times takes about 45 months. This is an estimate, not a promise. Re-check it every quarter with real numbers from your monitoring (Data Observability & Monitoring).

You pay for machines for as long as they run. So cost is about workers × time × price. Speed is not free, and the model from earlier shows why.

Take the 60-minute job and a made-up price of 0.50 per worker-hour. Total worker-minutes is workers × minutes. Look at what each size costs:

Cost is worker-minutes ÷ 60 × 0.50, so 78 worker-minutes cost 0.65. Going from 10 to 100 workers saves 4.86 minutes and costs 4.50 more. Each extra worker spends more time idle, waiting for the part that cannot be shared.

So buy speed only when someone needs it, such as a deadline. Pick the smallest cluster that meets the deadline with some spare time.

Doing less work (incremental loads, fewer columns, bigger files) is the cheapest speed-up. Warehouse Performance & Cost covers the warehouse side of the bill.

Build each scenario on the pipeline canvas with the chapter closed. The review tells you which requirement your design missed.