Working with Different Data Types

A column has one type: text, number, date, or a nested type such as a list. This chapter shows the functions for each type.

All of them are in pyspark.sql.functions, imported as F. They run inside Spark, on the executors. Chapter 14, UDFs & the Pandas API on Spark, shows why that is faster than your own Python function.

The examples use the shop: orders.csv from chapter 3, the customers, and the order items.

A partner app sends customer names with extra spaces and mixed case. This made-up data shows the cleaning functions:

from pyspark.sql import functions as F

partner = spark.createDataFrame([(" Asha ",), ("BEN",), ("carla\t",)], ["raw"])
partner.select("raw", F.initcap(F.trim("raw")).alias("clean")).show()

trim removed the spaces, but not the tab after carla. In Spark, trim removes only space characters.

regexp_replace changes every match of a pattern. Turn all whitespace into single spaces first, then trim:

partner.select(F.initcap(F.trim(F.regexp_replace("raw", r"\s+", " "))).alias("clean")).show()

regexp_extract pulls one part out of a text. When nothing matches, it returns an empty text, not NULL:

notes = spark.createDataFrame([("refund for order=103",), ("no order",)], ["note"])
notes.select("note", F.regexp_extract("note", r"order=(\d+)", 1).alias("order_id")).show()

A date is a day, such as 2026-01-03. A timestamp is a day and a time. inferSchema read order_date as a date.

Date functions add days, count days between dates, and format dates as text:

orders.select(
    "order_id", "order_date",
    F.date_add("order_date", 30).alias("due"),
    F.datediff(F.lit("2026-01-22").cast("date"), "order_date").alias("age_days"),
    F.date_format("order_date", "MMM d").alias("label"),
).show()

to_date turns text into a date. You give the format, such as dd/MM/yyyy for day/month/year. Text in another format fails in Spark 4:

raw = spark.createDataFrame([("03/01/2026",), ("2026-01-03",)], ["d"])
raw.select("d", F.to_date("d", "dd/MM/yyyy").alias("day")).show()

With F.try_to_date in the same code, the first row gives 2026-01-03 and the second gives NULL.

Timestamps also depend on the session time zone, the setting spark.sql.session.timeZone. Set it on purpose, for example to UTC, so a day starts at the same moment on every machine.

NULL means a missing value. Most functions skip it without a warning. That changes counts and averages:

orders.select(
    F.count("*").alias("rows"),
    F.count("amount").alias("amounts"),
    F.sum("amount").alias("total"),
    F.avg("amount").alias("avg"),
).show()

Two dashboards that disagree on a count are often this: one uses count(*), the other count of a column.

Decide what a NULL should become. fillna gives it a value, and dropna removes the row:

orders.fillna({"amount": 0.0}).select("order_id", "amount").filter("order_id >= 104").show()
print(orders.dropna(subset=["amount"]).count())

You cannot find NULL with ==. The test col("country") == None returns no rows at all. Use isNull(), or eqNullSafe, which treats two NULL values as equal:

customers.filter(F.col("country").eqNullSafe(None)).show()

A column can hold more than one value. An array is a list, a map holds key and value pairs, and a struct is a small record with named fields.

collect_list gathers values into an array. Here each order gets the list of its products:

baskets = (order_items
           .groupBy("order_id")
           .agg(F.sort_array(F.collect_list("product")).alias("products")))
baskets.show(truncate=False)

explode does the opposite. It makes one row for each item in the array:

baskets.select("order_id", F.explode("products").alias("product")).show()

Other array functions: F.size gives the length, and F.array_contains("products", "Lamp") is true for orders 103 and 106.

A struct groups columns into one. F.struct("status", "amount").alias("info") makes a column info, and you read a field with info.amount.