Openingloading the content…

Loadingone moment…

PySpark

Deploying Spark: spark-submit, YARN & Kubernetes

Submitting and sizing apps: client vs cluster mode, executor/core/memory config, dynamic allocation, and YARN vs Kubernetes.

⏱ 14 min readTopics chapter readerLevel · Production

01 · Orientation

Start here

⏱ 6 min · Topic 1 of 10

You can write a Spark job. Now you must run it on a real cluster, and ask for the right amount of machines.

You already know three words.

The driver is the program that plans the job.
Executors are the processes that run the tasks.
The cluster manager owns the machines and hands them out.

In "Running External Compute: Spark, Docker & Kubernetes Pods", an Airflow task only starts the job and waits. This chapter shows how that start works.

The chapter has eight parts.

Submit one small job with spark-submit.
Choose client or cluster mode: where the driver runs.
Send your Python files and libraries with the job.
Run on YARN, an older cluster manager.
Run on Kubernetes, another cluster manager.
Let Spark add and remove executors.
Check that your request fits the cluster.
Compare YARN and Kubernetes.

The cluster is the made-up one from "Spark Configurations": 10 nodes, each with 16 cores and 64 GB. Hosts, names and ids in this chapter are made up too.

02 · The command

Submit one small job

⏱ 6 min · Topic 2 of 10

spark-submit is the command that starts a Spark application. You give it your file, the cluster manager to use, and the resources to ask for.

We use a small job. It reads the shop orders and adds up the revenue of paid orders for each day. The file name is made up.

The job: orders_daily_job.pyworked example

SQL

Input data

orders (6 rows)6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

import sys
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders_daily").getOrCreate()
orders = spark.read.parquet(sys.argv[1])      # the input path comes from the command

daily = (orders.filter(F.col("status") == "paid")
               .groupBy("order_date")
               .agg(F.sum("amount").alias("revenue"))
               .orderBy("order_date"))
daily.show()

daily.show() prints (4 rows)

order_date	revenue
2026-01-03	40.00
2026-01-10	25.00
2026-01-15	15.50
2026-01-21	80.00

The paid orders are 101, 102, 104 and 106. Each one has its own date, so each gets a row. Orders 103 (refunded) and 105 (pending) are left out.

This is the whole job. Every way of submitting it in this chapter runs this same file.

Now run it on your own machine. The master is the cluster manager to use. The value local[*] means: run everything in one process, with one thread for each core.

Run it on your machineworked example

SQL

spark-submit --master "local[*]" orders_daily_job.py /data/orders

Options come first, then the file, then the values for your script. /data/orders is a made-up path. The output is the same four rows.

Local mode is for tests. For a real cluster, you change the master. This table lists the values and the options you will use most.

Option	What it sets
--master local[*]	Run on this machine only.
--master yarn	Run on a YARN cluster. The cluster address comes from the YARN config files on your machine.
--master k8s://https://HOST:PORT	Run on Kubernetes. HOST:PORT is the address of the Kubernetes API server, the program that controls the cluster.
--deploy-mode	Where the driver runs: client or cluster. The next part explains it.
--num-executors	How many executors to ask for. It sets spark.executor.instances.
--executor-cores, --executor-memory	The size of each executor.
--driver-memory	The memory of the driver.
--conf key=value	Any other Spark setting.
--name	The name of the application, shown in the UI.

Watch out

Put every spark-submit option before the file name. Anything after the file name goes to your script, so Spark never sees --executor-memory 19g there.

03 · Where the driver runs

Where the driver runs

⏱ 8 min · Topic 3 of 10

The deploy mode says where the driver runs. The executors always run in the cluster.

In client mode, the driver runs inside the spark-submit process, on the machine where you typed the command. In cluster mode, the cluster starts the driver on one of its own machines.

Here is the same job in both modes, on YARN.

The same job in both modesworked example

SQL

spark-submit --master yarn --deploy-mode client  orders_daily_job.py /data/orders
spark-submit --master yarn --deploy-mode cluster orders_daily_job.py /data/orders

Only --deploy-mode changes. The job code does not change.

The table shows what is different. Look at the last two rows: they matter for real jobs.

	Client mode	Cluster mode
Driver runs	On the machine that ran spark-submit	Inside the cluster
daily.show() prints	In your terminal	In the driver log, not in your terminal
You close the laptop	The driver dies and the job stops	The job keeps running
The network	The executors must reach your machine	The driver and executors are close together

This matters when Airflow starts the job. Airflow runs spark-submit on a worker.

In client mode, the driver lives on that worker. If the worker restarts, the job dies.

In cluster mode, spark-submit only sends the job and watches it. If that watching process dies, the job still runs. Then Airflow may not know the result, so make the job safe to run again.

Use client mode for notebooks and quick tests. Use cluster mode for scheduled jobs.

Interview question

Question: "Where does the driver run in each deploy mode?" Answer: client mode, in the spark-submit process. Cluster mode, on a machine inside the cluster. The YARN and Kubernetes parts show exactly where.

04 · Files and libraries

Send files and libraries

⏱ 8 min · Topic 4 of 10

On a cluster, your executors run on other machines. They do not have your files or your Python libraries. spark-submit can copy them for you.

This table lists the options. Maven Central, in the last row, is the main public store of Java libraries.

Option	What it sends	Example
--py-files	Python code: .py files, .zip files or .egg files, added to the Python path.	A zip of your helper modules
--files	Small files, copied to the working folder of every executor.	A settings file
--archives	An archive that Spark unpacks on each machine. Add #name to pick the folder name.	Your Python and its libraries, in one file
--jars	Java library files (.jar) from paths you give.	A database driver .jar
--packages	Java libraries from Maven Central, by name (group:name:version). Spark downloads them when the job starts. The version must match your Spark.	org.apache.spark:spark-avro_2.12:3.5.0

A Python library such as pandas needs more. Every machine needs the same Python and the same library versions. A virtual environment is a folder with one Python and its libraries.

On YARN, you pack a virtual environment into one file and send it with --archives.

The first block builds the file. The second block submits the job with it.

Pack an environment and send itworked example

SQL

# 1. Build the archive once (on a machine like the cluster).
python -m venv pyspark_venv
source pyspark_venv/bin/activate
pip install pandas pyarrow venv-pack
venv-pack -o pyspark_venv.tar.gz

# 2. Submit. The archive is unpacked as ./environment on every machine.
spark-submit --master yarn --deploy-mode cluster \
  --archives pyspark_venv.tar.gz#environment \
  --conf spark.pyspark.python=./environment/bin/python \
  --py-files helpers.zip \
  orders_daily_job.py /data/orders

The names pyspark_venv and helpers.zip are made up. The Spark guide "Python Package Management" sets the same thing with PYSPARK_PYTHON=./environment/bin/python.

On Kubernetes, the usual way is different. An image is one packed file with a program and everything it needs to run. Docker is the common tool that builds images.

You put Python, your libraries and your job file into one image. Every driver and executor starts from that image, so they all match. This is called baking them into the image.

Watch out

Do not use --packages for a production job unless you can accept a download at every start. If the download site is down, the job cannot start. A .jar you control, or a baked image, is safer.

05 · Hadoop clusters

Run on YARN

⏱ 9 min · Topic 5 of 10

YARN is the cluster manager of Hadoop, an older big-data system. Many company-owned clusters and some cloud services use it.

Five words explain how it works. The table lists them.

Word	Meaning
ResourceManager	The one boss. It knows what is free on every node and decides who gets it.
NodeManager	A helper on each worker node. It starts and watches containers.
Container	A bundle of cores and memory on one node. Each executor runs in one container.
ApplicationMaster	The first container of your application. It asks the ResourceManager for more containers.
Queue	A share of the cluster. Teams submit to a queue, for example --queue etl.

This is what happens in cluster mode, step by step.

spark-submit sends your application to the ResourceManager.
The ResourceManager starts the ApplicationMaster in a container on one node.
Your driver runs inside that ApplicationMaster container.
The driver asks for executor containers. The ResourceManager grants them.
The NodeManagers start the executors, and the executors connect to the driver.

In client mode, the driver stays on your machine. The ApplicationMaster only asks for executors.

Now submit the job with the sizes we will check later. The application id in the output is made up.

Submit to YARN in cluster modeworked example

SQL

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name orders_daily \
  --queue etl \
  --num-executors 29 \
  --executor-cores 5 \
  --executor-memory 19g \
  --driver-memory 4g \
  orders_daily_job.py /data/orders

spark-submit prints these lines (shortened, made-up id)

line
Submitted application application_1760000000000_0440
Application report for application_1760000000000_0440 (state: ACCEPTED)
Application report for application_1760000000000_0440 (state: RUNNING)
Application report for application_1760000000000_0440 (state: FINISHED)

ACCEPTED means YARN took the application but has not given it containers yet. RUNNING means the driver is working. FINISHED means it ended; the final status says succeeded or failed.

By default spark-submit stays open and prints the state, until the application ends.

If the state stays ACCEPTED, YARN cannot place the ApplicationMaster yet. The queue may have no room. Wait, or ask for less.

Read the driver log with the application id.

Read the logsworked example

SQL

yarn logs -applicationId application_1760000000000_0440

The driver log contains this table (4 rows)

order_date	revenue
2026-01-03	40.00
2026-01-10	25.00
2026-01-15	15.50
2026-01-21	80.00

The command prints the logs of every container. The driver output is in the log of the ApplicationMaster container. It works when log aggregation is on: YARN collects the logs in one place after the job.

Without log aggregation, the logs live on the worker nodes, and you must look for them there.

Watch out

A container holds the executor memory plus overhead: extra memory for work outside Spark’s own memory, such as Python. By default the overhead is 10% of the executor memory, at least 384 MB. If the container uses more, YARN kills it for exceeding memory limits, so raise spark.executor.memoryOverhead.

06 · Container clusters

Run on Kubernetes

⏱ 10 min · Topic 6 of 10

Kubernetes runs programs in containers. A container is a small, separate box that runs one program. It is not the same as a YARN container.

A pod is the unit Kubernetes starts, here one container. In cluster mode, the driver is one pod and each executor is one pod. You need four things: the API server address, an image, a namespace and a service account.

The API server address is the master: k8s://https://HOST:PORT.
The image is a Docker image with Spark, Python, your libraries and your job file.
A namespace is a named area in the cluster, for example spark-jobs.
A service account is an identity. The driver uses it to create executor pods, so it needs that permission.

This is what happens in cluster mode. Spark on Kubernetes has no ApplicationMaster.

spark-submit asks the API server to create the driver pod.
The driver pod starts and asks the API server for executor pods.
The executor pods start and connect to the driver.
When the job ends, Spark deletes the executor pods. The driver pod stays in the state Completed, so you can read its logs.

Here is the job. The host, image and names are made up. The prefix local:// means the file is inside the image.

Submit to Kubernetesworked example

SQL

spark-submit \
  --master k8s://https://k8s-api.example.com:6443 \
  --deploy-mode cluster \
  --name orders-daily \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.container.image=registry.example.com/shop/orders-spark:1.4 \
  --conf spark.kubernetes.driver.pod.name=orders-daily-driver \
  --conf spark.executor.instances=27 \
  --conf spark.executor.cores=5 \
  --conf spark.kubernetes.executor.request.cores=4.5 \
  --conf spark.executor.memory=18g \
  --conf spark.executor.memoryOverhead=2g \
  --conf spark.driver.memory=4g \
  --conf spark.driver.memoryOverhead=1g \
  local:///opt/app/orders_daily_job.py /data/orders

Compare with YARN: the master, the image, the namespace and the account are new.

Look at the pods with kubectl, the Kubernetes command line tool. The table shows the pods during the run and after it. Only the first executor is shown.

When	Pod	Status
During the run	orders-daily-driver	Running
During the run	orders-daily-440-exec-1 (and 26 more)	Running
After the run	orders-daily-driver	Completed
After the run	No executor pods	Spark deleted them

After the run, read the driver log with kubectl logs orders-daily-driver -n spark-jobs. The four result rows are there.

Two settings look alike. spark.executor.cores is how many tasks one executor runs at the same time. spark.kubernetes.executor.request.cores is how much CPU the pod asks Kubernetes for.

Here the executors run 5 tasks each but ask for 4.5 cores. This leaves room for small system pods on each node. If you do not set the request, it equals spark.executor.cores.

Watch out

For PySpark jobs, Spark 3.x on Kubernetes uses a bigger default memory overhead: 40%, not 10%. An executor of 19 GB would then ask for 26.6 GB. Set the overhead yourself, and check your version.

Two more tools exist. A pod template is a file that sets extra pod details, such as which nodes to use and extra disks. The Spark operator is a separate add-on that starts jobs from a short settings file you give Kubernetes.

07 · Flexible size

Add and remove executors

⏱ 6 min · Topic 7 of 10

You met dynamic allocation in "Spark Configurations". Spark adds executors when tasks wait, and removes them when they are idle. It is off by default.

The new question is how it stays safe on each cluster manager. A removed executor must not take its shuffle files, the data that later tasks still read. The table shows the choices.

	YARN	Kubernetes
Shuffle tracking	Works. Spark keeps an executor while its shuffle files are still needed. Set spark.dynamicAllocation.shuffleTracking.enabled=true.	The usual choice. Set the same setting.
External shuffle service	Your cluster team runs a separate service on each NodeManager. It keeps serving shuffle files after an executor is removed. Set spark.shuffle.service.enabled=true.	Spark does not provide one for Kubernetes.

Here is the Kubernetes job with dynamic allocation. The limit 27 is the number we will check in the next part.

Dynamic allocation on Kubernetesworked example

SQL

  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=2 \
  --conf spark.dynamicAllocation.maxExecutors=27 \

Add these lines to the Kubernetes command. The job now runs with 2 to 27 executors.

An executor that has no tasks for 60 seconds is removed by default.

If you also set --num-executors, Spark starts with that many and then adjusts.

Watch out

Always set maxExecutors. By default there is no limit, so one big job can take the whole cluster, and other jobs wait.

08 · Fit and choice

Sizing and cluster managers

⏱ 8 min · Topic 8 of 10

Before you submit, check that your request fits the cluster. Our cluster has 10 nodes. We assume each node offers 15 cores and 63 GB to Spark.

A pod also holds the executor memory plus the overhead, like a YARN container. The table does the arithmetic for both commands from this chapter.

Check	YARN command	Kubernetes command
Executors	29	27
Memory of one container or pod	19 GB + 1.9 GB overhead = 20.9 GB	18 GB + 2 GB overhead = 20 GB
Executors on one node	3 × 20.9 = 62.7 GB, under 63	3 × 20 = 60 GB, under 63. The rest is for system pods.
Driver	4 GB + about 0.4 GB = 4.4 GB, inside the ApplicationMaster	4 GB + 1 GB = 5 GB, in its own pod
All memory asked	29 × 20.9 + 4.4 = 610.5 GB of 630 GB	27 × 20 + 5 = 545 GB of 630 GB
Tasks at once	29 × 5 = 145	27 × 5 = 135

Read the YARN column. One node holds the driver. It has room for only 2 executors: 41.8 + 4.4 = 46.2 GB.

With a third executor, that node would need 67.1 GB, more than 63. So 9 nodes hold 3 each, and one holds 2: 27 + 2 = 29.

YARN rounds each container up to a whole GB, so 20.9 becomes 21. It still fits: 3 × 21 = 63. The driver node fits too: 2 × 21 + 5 = 47.

Read the Kubernetes column. 9 nodes hold 3 executors each, and the driver pod sits on the 10th node. That node has room for 2 more executors, but we keep it for the driver and other pods.

Now compare the two managers.

	YARN	Kubernetes
Common in	Hadoop clusters and some cloud services	Cloud platforms that already run containers
Driver in cluster mode	Inside the ApplicationMaster	In its own pod
Libraries and Python	Send with --archives and --py-files	Bake into the Docker image
Share the cluster	Queues	Namespaces, with quotas set by the platform team
Logs	yarn logs -applicationId ID	kubectl logs DRIVER_POD

The choice is usually made by your company, not by you. Know both, and know the settings that change.

Watch out

A request above what a node offers never starts. On YARN, spark-submit fails at once if a container is above the YARN maximum. On Kubernetes the pod stays Pending.

Openingloading the content…

PySpark

Deploying Spark: spark-submit, YARN & Kubernetes

Submitting and sizing apps: client vs cluster mode, executor/core/memory config, dynamic allocation, and YARN vs Kubernetes.

⏱ 14 min readTopics chapter readerLevel · Production

01 · Orientation

Start here

⏱ 6 min · Topic 1 of 10

You can write a Spark job. Now you must run it on a real cluster, and ask for the right amount of machines.

You already know three words.

The driver is the program that plans the job.
Executors are the processes that run the tasks.
The cluster manager owns the machines and hands them out.

In "Running External Compute: Spark, Docker & Kubernetes Pods", an Airflow task only starts the job and waits. This chapter shows how that start works.

The chapter has eight parts.

Submit one small job with spark-submit.
Choose client or cluster mode: where the driver runs.
Send your Python files and libraries with the job.
Run on YARN, an older cluster manager.
Run on Kubernetes, another cluster manager.
Let Spark add and remove executors.
Check that your request fits the cluster.
Compare YARN and Kubernetes.

The cluster is the made-up one from "Spark Configurations": 10 nodes, each with 16 cores and 64 GB. Hosts, names and ids in this chapter are made up too.

02 · The command

Submit one small job

⏱ 6 min · Topic 2 of 10

spark-submit is the command that starts a Spark application. You give it your file, the cluster manager to use, and the resources to ask for.

We use a small job. It reads the shop orders and adds up the revenue of paid orders for each day. The file name is made up.

The job: orders_daily_job.pyworked example

SQL

Input data

orders (6 rows)6 rows

order_id	customer_id	status	amount	order_date
101	1	paid	40.00	2026-01-03
102	1	paid	25.00	2026-01-10
103	2	refunded	60.00	2026-01-11
104	3	paid	15.50	2026-01-15
105	3	pending	NULL	2026-01-20
106	4	paid	80.00	2026-01-21

import sys
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders_daily").getOrCreate()
orders = spark.read.parquet(sys.argv[1])      # the input path comes from the command

daily = (orders.filter(F.col("status") == "paid")
               .groupBy("order_date")
               .agg(F.sum("amount").alias("revenue"))
               .orderBy("order_date"))
daily.show()

daily.show() prints (4 rows)

order_date	revenue
2026-01-03	40.00
2026-01-10	25.00
2026-01-15	15.50
2026-01-21	80.00

The paid orders are 101, 102, 104 and 106. Each one has its own date, so each gets a row. Orders 103 (refunded) and 105 (pending) are left out.

This is the whole job. Every way of submitting it in this chapter runs this same file.

Now run it on your own machine. The master is the cluster manager to use. The value local[*] means: run everything in one process, with one thread for each core.

Run it on your machineworked example

SQL

spark-submit --master "local[*]" orders_daily_job.py /data/orders

Options come first, then the file, then the values for your script. /data/orders is a made-up path. The output is the same four rows.

Local mode is for tests. For a real cluster, you change the master. This table lists the values and the options you will use most.

Option	What it sets
--master local[*]	Run on this machine only.
--master yarn	Run on a YARN cluster. The cluster address comes from the YARN config files on your machine.
--master k8s://https://HOST:PORT	Run on Kubernetes. HOST:PORT is the address of the Kubernetes API server, the program that controls the cluster.
--deploy-mode	Where the driver runs: client or cluster. The next part explains it.
--num-executors	How many executors to ask for. It sets spark.executor.instances.
--executor-cores, --executor-memory	The size of each executor.
--driver-memory	The memory of the driver.
--conf key=value	Any other Spark setting.
--name	The name of the application, shown in the UI.

Watch out

Put every spark-submit option before the file name. Anything after the file name goes to your script, so Spark never sees --executor-memory 19g there.

03 · Where the driver runs

Where the driver runs

⏱ 8 min · Topic 3 of 10

The deploy mode says where the driver runs. The executors always run in the cluster.

In client mode, the driver runs inside the spark-submit process, on the machine where you typed the command. In cluster mode, the cluster starts the driver on one of its own machines.

Here is the same job in both modes, on YARN.

The same job in both modesworked example

SQL

spark-submit --master yarn --deploy-mode client  orders_daily_job.py /data/orders
spark-submit --master yarn --deploy-mode cluster orders_daily_job.py /data/orders

Only --deploy-mode changes. The job code does not change.

The table shows what is different. Look at the last two rows: they matter for real jobs.

	Client mode	Cluster mode
Driver runs	On the machine that ran spark-submit	Inside the cluster
daily.show() prints	In your terminal	In the driver log, not in your terminal
You close the laptop	The driver dies and the job stops	The job keeps running
The network	The executors must reach your machine	The driver and executors are close together

This matters when Airflow starts the job. Airflow runs spark-submit on a worker.

In client mode, the driver lives on that worker. If the worker restarts, the job dies.

In cluster mode, spark-submit only sends the job and watches it. If that watching process dies, the job still runs. Then Airflow may not know the result, so make the job safe to run again.

Use client mode for notebooks and quick tests. Use cluster mode for scheduled jobs.

Interview question

04 · Files and libraries

Send files and libraries

⏱ 8 min · Topic 4 of 10

On a cluster, your executors run on other machines. They do not have your files or your Python libraries. spark-submit can copy them for you.

This table lists the options. Maven Central, in the last row, is the main public store of Java libraries.

Option	What it sends	Example
--py-files	Python code: .py files, .zip files or .egg files, added to the Python path.	A zip of your helper modules
--files	Small files, copied to the working folder of every executor.	A settings file
--archives	An archive that Spark unpacks on each machine. Add #name to pick the folder name.	Your Python and its libraries, in one file
--jars	Java library files (.jar) from paths you give.	A database driver .jar
--packages	Java libraries from Maven Central, by name (group:name:version). Spark downloads them when the job starts. The version must match your Spark.	org.apache.spark:spark-avro_2.12:3.5.0

A Python library such as pandas needs more. Every machine needs the same Python and the same library versions. A virtual environment is a folder with one Python and its libraries.

On YARN, you pack a virtual environment into one file and send it with --archives.

The first block builds the file. The second block submits the job with it.

Pack an environment and send itworked example

SQL

# 1. Build the archive once (on a machine like the cluster).
python -m venv pyspark_venv
source pyspark_venv/bin/activate
pip install pandas pyarrow venv-pack
venv-pack -o pyspark_venv.tar.gz

# 2. Submit. The archive is unpacked as ./environment on every machine.
spark-submit --master yarn --deploy-mode cluster \
  --archives pyspark_venv.tar.gz#environment \
  --conf spark.pyspark.python=./environment/bin/python \
  --py-files helpers.zip \
  orders_daily_job.py /data/orders

The names pyspark_venv and helpers.zip are made up. The Spark guide "Python Package Management" sets the same thing with PYSPARK_PYTHON=./environment/bin/python.

On Kubernetes, the usual way is different. An image is one packed file with a program and everything it needs to run. Docker is the common tool that builds images.

You put Python, your libraries and your job file into one image. Every driver and executor starts from that image, so they all match. This is called baking them into the image.

Watch out

Do not use --packages for a production job unless you can accept a download at every start. If the download site is down, the job cannot start. A .jar you control, or a baked image, is safer.

05 · Hadoop clusters

Run on YARN

⏱ 9 min · Topic 5 of 10

YARN is the cluster manager of Hadoop, an older big-data system. Many company-owned clusters and some cloud services use it.

Five words explain how it works. The table lists them.

Word	Meaning
ResourceManager	The one boss. It knows what is free on every node and decides who gets it.
NodeManager	A helper on each worker node. It starts and watches containers.
Container	A bundle of cores and memory on one node. Each executor runs in one container.
ApplicationMaster	The first container of your application. It asks the ResourceManager for more containers.
Queue	A share of the cluster. Teams submit to a queue, for example --queue etl.

This is what happens in cluster mode, step by step.

spark-submit sends your application to the ResourceManager.
The ResourceManager starts the ApplicationMaster in a container on one node.
Your driver runs inside that ApplicationMaster container.
The driver asks for executor containers. The ResourceManager grants them.
The NodeManagers start the executors, and the executors connect to the driver.

In client mode, the driver stays on your machine. The ApplicationMaster only asks for executors.

Now submit the job with the sizes we will check later. The application id in the output is made up.

Submit to YARN in cluster modeworked example

SQL

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name orders_daily \
  --queue etl \
  --num-executors 29 \
  --executor-cores 5 \
  --executor-memory 19g \
  --driver-memory 4g \
  orders_daily_job.py /data/orders

spark-submit prints these lines (shortened, made-up id)

line
Submitted application application_1760000000000_0440
Application report for application_1760000000000_0440 (state: ACCEPTED)
Application report for application_1760000000000_0440 (state: RUNNING)
Application report for application_1760000000000_0440 (state: FINISHED)

ACCEPTED means YARN took the application but has not given it containers yet. RUNNING means the driver is working. FINISHED means it ended; the final status says succeeded or failed.

By default spark-submit stays open and prints the state, until the application ends.

If the state stays ACCEPTED, YARN cannot place the ApplicationMaster yet. The queue may have no room. Wait, or ask for less.

Read the driver log with the application id.

Read the logsworked example

SQL

yarn logs -applicationId application_1760000000000_0440

The driver log contains this table (4 rows)

order_date	revenue
2026-01-03	40.00
2026-01-10	25.00
2026-01-15	15.50
2026-01-21	80.00

Without log aggregation, the logs live on the worker nodes, and you must look for them there.

Watch out

06 · Container clusters

Run on Kubernetes

⏱ 10 min · Topic 6 of 10

Kubernetes runs programs in containers. A container is a small, separate box that runs one program. It is not the same as a YARN container.

The API server address is the master: k8s://https://HOST:PORT.
The image is a Docker image with Spark, Python, your libraries and your job file.
A namespace is a named area in the cluster, for example spark-jobs.
A service account is an identity. The driver uses it to create executor pods, so it needs that permission.

This is what happens in cluster mode. Spark on Kubernetes has no ApplicationMaster.

spark-submit asks the API server to create the driver pod.
The driver pod starts and asks the API server for executor pods.
The executor pods start and connect to the driver.
When the job ends, Spark deletes the executor pods. The driver pod stays in the state Completed, so you can read its logs.

Here is the job. The host, image and names are made up. The prefix local:// means the file is inside the image.

Submit to Kubernetesworked example

SQL

spark-submit \
  --master k8s://https://k8s-api.example.com:6443 \
  --deploy-mode cluster \
  --name orders-daily \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.container.image=registry.example.com/shop/orders-spark:1.4 \
  --conf spark.kubernetes.driver.pod.name=orders-daily-driver \
  --conf spark.executor.instances=27 \
  --conf spark.executor.cores=5 \
  --conf spark.kubernetes.executor.request.cores=4.5 \
  --conf spark.executor.memory=18g \
  --conf spark.executor.memoryOverhead=2g \
  --conf spark.driver.memory=4g \
  --conf spark.driver.memoryOverhead=1g \
  local:///opt/app/orders_daily_job.py /data/orders

Compare with YARN: the master, the image, the namespace and the account are new.

Look at the pods with kubectl, the Kubernetes command line tool. The table shows the pods during the run and after it. Only the first executor is shown.

When	Pod	Status
During the run	orders-daily-driver	Running
During the run	orders-daily-440-exec-1 (and 26 more)	Running
After the run	orders-daily-driver	Completed
After the run	No executor pods	Spark deleted them

After the run, read the driver log with kubectl logs orders-daily-driver -n spark-jobs. The four result rows are there.

Two settings look alike. spark.executor.cores is how many tasks one executor runs at the same time. spark.kubernetes.executor.request.cores is how much CPU the pod asks Kubernetes for.

Here the executors run 5 tasks each but ask for 4.5 cores. This leaves room for small system pods on each node. If you do not set the request, it equals spark.executor.cores.

Watch out

For PySpark jobs, Spark 3.x on Kubernetes uses a bigger default memory overhead: 40%, not 10%. An executor of 19 GB would then ask for 26.6 GB. Set the overhead yourself, and check your version.

07 · Flexible size

Add and remove executors

⏱ 6 min · Topic 7 of 10

You met dynamic allocation in "Spark Configurations". Spark adds executors when tasks wait, and removes them when they are idle. It is off by default.

The new question is how it stays safe on each cluster manager. A removed executor must not take its shuffle files, the data that later tasks still read. The table shows the choices.

	YARN	Kubernetes
Shuffle tracking	Works. Spark keeps an executor while its shuffle files are still needed. Set spark.dynamicAllocation.shuffleTracking.enabled=true.	The usual choice. Set the same setting.
External shuffle service	Your cluster team runs a separate service on each NodeManager. It keeps serving shuffle files after an executor is removed. Set spark.shuffle.service.enabled=true.	Spark does not provide one for Kubernetes.

Here is the Kubernetes job with dynamic allocation. The limit 27 is the number we will check in the next part.

Dynamic allocation on Kubernetesworked example

SQL

  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=2 \
  --conf spark.dynamicAllocation.maxExecutors=27 \

Add these lines to the Kubernetes command. The job now runs with 2 to 27 executors.

An executor that has no tasks for 60 seconds is removed by default.

If you also set --num-executors, Spark starts with that many and then adjusts.

Watch out

Always set maxExecutors. By default there is no limit, so one big job can take the whole cluster, and other jobs wait.

08 · Fit and choice

Sizing and cluster managers

⏱ 8 min · Topic 8 of 10

Before you submit, check that your request fits the cluster. Our cluster has 10 nodes. We assume each node offers 15 cores and 63 GB to Spark.

A pod also holds the executor memory plus the overhead, like a YARN container. The table does the arithmetic for both commands from this chapter.

Check	YARN command	Kubernetes command
Executors	29	27
Memory of one container or pod	19 GB + 1.9 GB overhead = 20.9 GB	18 GB + 2 GB overhead = 20 GB
Executors on one node	3 × 20.9 = 62.7 GB, under 63	3 × 20 = 60 GB, under 63. The rest is for system pods.
Driver	4 GB + about 0.4 GB = 4.4 GB, inside the ApplicationMaster	4 GB + 1 GB = 5 GB, in its own pod
All memory asked	29 × 20.9 + 4.4 = 610.5 GB of 630 GB	27 × 20 + 5 = 545 GB of 630 GB
Tasks at once	29 × 5 = 145	27 × 5 = 135

Read the YARN column. One node holds the driver. It has room for only 2 executors: 41.8 + 4.4 = 46.2 GB.

With a third executor, that node would need 67.1 GB, more than 63. So 9 nodes hold 3 each, and one holds 2: 27 + 2 = 29.

YARN rounds each container up to a whole GB, so 20.9 becomes 21. It still fits: 3 × 21 = 63. The driver node fits too: 2 × 21 + 5 = 47.

Read the Kubernetes column. 9 nodes hold 3 executors each, and the driver pod sits on the 10th node. That node has room for 2 more executors, but we keep it for the driver and other pods.

Now compare the two managers.

	YARN	Kubernetes
Common in	Hadoop clusters and some cloud services	Cloud platforms that already run containers
Driver in cluster mode	Inside the ApplicationMaster	In its own pod
Libraries and Python	Send with --archives and --py-files	Bake into the Docker image
Share the cluster	Queues	Namespaces, with quotas set by the platform team
Logs	yarn logs -applicationId ID	kubectl logs DRIVER_POD

The choice is usually made by your company, not by you. Know both, and know the settings that change.

Watch out

A request above what a node offers never starts. On YARN, spark-submit fails at once if a container is above the YARN maximum. On Kubernetes the pod stays Pending.

Deploying Spark: spark-submit, YARN & Kubernetes

Start here

Submit one small job

Where the driver runs

Send files and libraries

Run on YARN

Run on Kubernetes

Add and remove executors

Sizing and cluster managers

Check yourself

Next Chapter

Deploying Spark: spark-submit, YARN & Kubernetes

Start here

Submit one small job

Where the driver runs

Send files and libraries

Run on YARN

Run on Kubernetes

Add and remove executors

Sizing and cluster managers

Check yourself

Next Chapter