What You'll Master Here
Built-in functions run inside the JVM where Catalyst can optimise them. A plain Python UDF ships each row out to a Python worker and back, one at a time. The whole game is avoiding that round trip, or making it vectorised.
Sometimes the built-in functions are not enough and you need your own logic. That is what a UDF (user-defined function) is: a Python function you register so Spark can call it on every row. It is the escape hatch, and like most escape hatches, it is powerful and easy to misuse.
The catch is performance. A plain Python UDF is opaque to Catalyst (Chapter 13): the optimizer cannot see inside it, so no pushdown, no codegen, and every row crosses a slow Python boundary. This chapter shows when a UDF is justified, why the naive one is slow, and the faster alternatives, pandas/Arrow UDFs and the Pandas API on Spark.
You will leave knowing the right order of preference: built-in functions first, then a vectorised pandas UDF, and only then a plain Python UDF, with the reasoning to defend that choice in an interview.
Built-in functions run inside the JVM where Catalyst can optimise them. A plain Python UDF ships each row out to a Python worker and back, one at a time. The whole game is avoiding that round trip, or making it vectorised.
UDFs are the most common self-inflicted performance wound in Spark. Knowing when to avoid them, and which kind to reach for when you cannot, separates engineers who write correct-but-slow jobs from those who write fast ones.
- UDF
- A user-defined function: your own logic registered to run per row.
- built-in (native) function
- A function in pyspark.sql.functions that Catalyst can optimise and codegen.
- serialization boundary
- The cost of moving data between the JVM and Python for each UDF call.
- vectorised UDF
- A pandas/Arrow UDF that processes a batch of rows at once, not one at a time.
Reaching for a UDF before checking pyspark.sql.functions. You give up Catalyst optimisation for logic that often already exists as a fast native function.
Search the built-in functions before writing any UDF.
If you must write one, prefer a pandas/Arrow UDF over a plain Python UDF.
Treat a plain Python UDF as a last resort, and comment why it was unavoidable.
A UDF is a black box to the optimizer. Native functions are glass: Catalyst sees and optimises them. Prefer glass; reach for the black box only when you must, and make it vectorised.
UDFs let you run custom logic per row, but they cost optimisation and (for plain Python UDFs) speed. Prefer built-ins; when you cannot, use a vectorised pandas UDF.
- Why can Catalyst not optimise a plain Python UDF?
- What is the order of preference for custom logic in Spark?