The structured APIs: DataFrames and Datasets vs RDDs, schemas, columns, and typed vs untyped processing.
⏱ 19 min readTopics chapter readerLevel · Structured APIs
01 · Orientation
What You'll Master Here
it has named, typed columns (the schema) and its rows are split into partitions across the cluster.
⏱ 4 min · Topic 1 of 7
DataFrames are the API you will write all day in Spark. A DataFrame is a distributed table, rows and named, typed columns, that Spark understands well enough to optimise. This chapter makes you fluent in what a DataFrame is, how columns and expressions work, what a schema is, and how Datasets relate (briefly, since in PySpark you use DataFrames).
You already know why DataFrames beat RDDs (Chapter 4) and how laziness works (Chapter 5). Here we get concrete about the structured API itself: creating DataFrames, defining schemas, and referencing columns, the building blocks for every operation in Chapters 7–10.
Everything here comes with runnable PySpark, an explicit schema, and the resulting output, so you see exactly how rows transform.
Core mental model
A DataFrame is a spreadsheet that does not fit on one computer: it has named, typed columns (the schema) and its rows are split into partitions across the cluster.
Why it matters
The DataFrame + schema model is the foundation of Spark SQL, the Catalyst optimizer, and basically all production Spark. Getting comfortable here makes the rest of the module hands-on.
DataFrame
A distributed collection of rows with a schema (named, typed columns).
schema
The columns of a DataFrame and their data types.
Column
A reference to a column, used to build expressions (col("age") + 1).
Dataset
A JVM-typed version of a DataFrame (Scala/Java); in PySpark you use DataFrames.
Common mistake
Treating a DataFrame like a pandas DataFrame held on one machine. It is distributed and lazy; operations run across partitions only when an action fires.
Better habit
Always know your DataFrame's schema (printSchema()).
Reference columns with col() / expr() to build clear expressions.
Default to DataFrames; reach for Datasets only in typed Scala/Java code.
The big idea
A DataFrame = rows + a schema, distributed and lazy. The schema is what lets Spark optimise; the distribution is what lets it scale.
Remember this
A DataFrame is a distributed, lazy table with a schema. Columns and expressions build your logic; the schema is what makes Spark fast. In PySpark, DataFrame is the API you use.
Practice2 prompts
Define a DataFrame in one sentence using "schema" and "partitions".
Why is the schema central to Spark's ability to optimise?
02 · The structure
DataFrames: Distributed Tables with a Schema
One table, many partitions. You think in rows and columns; Spark executes in partitions and tasks.
⏱ 7 min · Topic 2 of 7
Logically, a DataFrame looks like a database table or a spreadsheet: rows of data with named columns, each column having a type (string, integer, timestamp...). That set of column names and types is the schema. Physically, those rows are split into partitions spread across the executors, the same distribution you saw for RDDs.
This dual nature, a single logical table that is physically distributed, is what makes DataFrames both easy to reason about and able to scale. You write table operations; Spark runs them in parallel across the partitions.
The first diagram shows both views: the logical table on one side, the physical partitions on the other. They are the same data, seen two ways.
Because the schema is known, Spark can turn your DataFrame code into an optimised execution plan before it runs a single row. Call df.explain() and Spark prints the physical plan it chose — the second figure shows one. The optimizer that produces that plan is called Catalyst; we name it here and dig into how it rewrites plans in the Spark SQL & Catalyst Optimizer chapter (Chapter 13). For now the point is simply: the schema is what makes a plan, and therefore optimisation, possible at all.
Core mental model
One table, many partitions. You think in rows and columns; Spark executes in partitions and tasks.
Why it matters
Holding both views, logical table and physical partitions, in your head is what lets you write clear DataFrame code and still reason about parallelism, shuffles, and skew.
StructType / StructField
The classes used to declare a DataFrame schema (columns and types).
createDataFrame()
Builds a DataFrame from local rows (and optionally a schema).
nullable
The third StructField argument: whether a column may contain nulls.
partition (DataFrame)
A slice of the DataFrame's rows processed by one task.
A DataFrame is one logical table (named, typed columns) that is physically split into partitions across the cluster. You operate on the table; Spark runs it on the partitions.
== Physical Plan ==
*(1) Project [name#0]
+- *(1) Filter (isnotnull(age#1) AND (age#1 >= 40))
+- *(1) Scan ExistingRDD[name#0,age#1,city#2]
people.filter(col('age') >= 40).select('name').explain() prints the physical plan in the SQL / DataFrame tab. The schema let the optimizer (Catalyst, covered in Chapter 13) prune to just the columns it needs and push the filter down to the scan — that is the payoff of the schema being known up front. See Chapter 3 for the full Spark UI tour.
Create a DataFrame with an explicit schemaworked example
printSchema() prints: name: string, age: integer, city: string — exactly the StructType we declared, no guessing.
StructType/StructField declare the schema up front. createDataFrame builds a distributed DataFrame from the rows and that schema. Declaring the schema (rather than inferring it) is faster and removes ambiguity, the production default.
Common mistake
Ignoring the schema until something breaks. Type surprises (numbers read as strings) cause silent wrong results; check printSchema() early.
Better habit
Declare explicit schemas for production reads (Chapter 7).
Confirm types with printSchema() after creating or reading data.
Think "table" for logic, "partitions" for performance.
Production reality
Explicit schemas are not just faster, they are a contract. They fail loudly when the data shape changes, instead of silently mis-typing a column.
Remember this
A DataFrame is a logical table with an explicit schema, physically split into partitions. Declare schemas with StructType for speed and safety; confirm them with printSchema().
Practice2 prompts
Write a StructType for (id: int, email: string, signup: timestamp).
Why is an explicit schema safer than inferSchema in production?
03 · Referencing data
Columns & Expressions
A Column is a formula, not a value. You assemble formulas; Spark compiles and runs them efficiently across every partition.
⏱ 4 min · Topic 3 of 7
You manipulate DataFrames through columns. A Column is a reference, not data, that you combine into expressions: col("amount") * 0.9, col("age") > 18, lower(col("name")). Spark turns these expressions into optimised code; it never runs them row by row in Python (unlike a UDF).
There are a few equivalent ways to name a column: col("x"), df["x"], df.x, or inside expr("x * 2") / SQL strings. They all build the same Column expression. Using col() and the built-in functions (from pyspark.sql.functions) keeps Spark in its fast, optimised path.
The key habit: prefer built-in column expressions over Python UDFs. Built-ins are understood and optimised by Catalyst and Tungsten; UDFs are a black box Spark cannot optimise (Chapter 14).
Core mental model
A Column is a formula, not a value. You assemble formulas; Spark compiles and runs them efficiently across every partition.
Why it matters
Almost every transformation, select, filter, withColumn, is built from column expressions. Writing them with built-ins (not UDFs) is one of the easiest, biggest performance wins in Spark.
col() / df["x"]
Ways to reference a column to build an expression.
expression
A computation over columns (arithmetic, comparison, function calls).
pyspark.sql.functions
The library of built-in column functions (upper, when, to_date, sum...).
withColumn() / select()
Add/replace a column / choose columns, both take column expressions.
Build column expressions with built-in functionsworked example
upper() and the comparison are built-in column expressions, optimised by Spark. No Python ran per row; Catalyst compiled the expressions.
col("city") references a column; upper(...) and col("age") >= 40 build expressions; withColumn adds them as new columns; select keeps what you want. Using built-in functions (not a Python UDF) keeps the whole thing in Spark's optimised engine.
Common mistake
Writing a Python UDF for something a built-in already does. UDFs are opaque to Catalyst and much slower; use built-ins whenever possible (Chapter 14).
Better habit
Build logic from col() + built-in functions.
Reach for a UDF only when no built-in exists.
Use withColumn to add/replace and select to project.
Interview note
"Why avoid UDFs?" Answer: "Built-in column functions are understood and optimised by Catalyst/Tungsten; a Python UDF is a black box Spark must call row by row, so it is slower and blocks optimization."
Remember this
Manipulate DataFrames through column expressions built from col() and built-in functions, which Spark optimises. Avoid Python UDFs when a built-in exists.
Practice2 prompts
Write an expression that flags rows where amount is above the average (conceptually).
Why is a built-in function faster than an equivalent Python UDF?
04 · The triad
DataFrame vs Dataset vs RDD
RDD = raw objects (foundation). DataFrame = a table with an optimizer (what you use in Python). Dataset = a typed table (Scala/Java only).
⏱ 4 min · Topic 4 of 7
Spark has three data abstractions, and it helps to see them as layers. RDDs are the low-level foundation (objects, no schema). DataFrames add a schema and the Catalyst optimizer (a DataFrame is really a Dataset of generic Row objects). Datasets add compile-time type safety — you work with typed JVM objects — but they exist only in Scala and Java. In PySpark, there is no separate Dataset API: Python is dynamically typed, so you use DataFrames for everything (a PySpark DataFrame is the equivalent of Scala's Dataset[Row] — in Scala it is literally type DataFrame = Dataset[Row]). That is why this whole module is DataFrame-first.
Be precise about what "type safety" means per abstraction, because this is the row people get wrong. An RDD has no schema and no structural checking at all: an RDD[Person] in Scala only has the host language's generic typing — the compiler knows the element is a Person object, not any knowledge of columns or their types — and in PySpark an RDD is completely untyped. A DataFrame is checked at plan-analysis time: misspell a column and Spark raises an AnalysisException when it analyses the query, before any data moves. That is earlier than failing mid-job, but it is still not compile-time. Only a Dataset gives true compile-time type safety, and only in Scala/Java.
A common myth is that Python DataFrames are much slower than Scala DataFrames. They are not. Both compile to the same Catalyst plan and execute on the JVM via Tungsten; your Python code does not run per row. The performance gap only appears when you drop to Python UDFs or RDD lambdas, where rows must be serialized across the JVM-to-Python boundary. Arrow-based pandas UDFs narrow even that gap (see the UDFs & pandas API chapter).
Core mental model
RDD = raw objects (foundation). DataFrame = a table with an optimizer (what you use in Python). Dataset = a typed table (Scala/Java only).
Why it matters
Interviewers love "DataFrame vs Dataset vs RDD?", and the type-safety row is exactly where weak answers fall apart (claiming RDDs are type-safe, or that Python DataFrames are slow). Knowing the layering precisely — and that PySpark uses DataFrames at Scala-DataFrame speed — shows you understand the API landscape, not just one corner of it.
Dataset[Row]
What a DataFrame is under the hood; a PySpark DataFrame is this.
type safety
Datasets catch type errors at compile time; DataFrames catch column/type errors at plan-analysis time (before execution); RDDs have no structural checking at all.
Catalyst
The optimizer DataFrames and Datasets share; RDDs do not have it.
encoder
The mechanism that maps typed JVM objects to Spark's internal format (Datasets, Scala/Java).
The three abstractions
Aspect
RDD
DataFrame
Dataset
Schema?
No
Yes
Yes
Optimizer (Catalyst)?
No
Yes
Yes
Type safety
None (no schema — no structural checking)
Runtime / plan analysis (column names + types checked when the query is analyzed)
Compile-time (typed JVM objects; Scala/Java only)
Languages
All
All
Scala / Java only
In PySpark you use
Rarely
Always
N/A (use DataFrame)
Common mistake
Looking for a typed Dataset API in PySpark. There isn't one, Python uses DataFrames; Datasets are a Scala/Java feature.
Better habit
In PySpark, reach for DataFrames by default.
Mention Datasets as the typed Scala/Java sibling when asked.
Remember the partitioned, fault-tolerant execution model that DataFrames and Datasets build on was introduced by RDDs — but say "build on", not "compile to the RDD API".
Interview note
"DataFrame vs Dataset vs RDD?" Answer: "RDD = no schema/optimizer; DataFrame = schema + Catalyst (a Dataset[Row]); Dataset = typed, Scala/Java only. In PySpark we use DataFrames."
Remember this
RDD (foundation) → DataFrame (schema + Catalyst) → Dataset (typed, Scala/Java). PySpark uses DataFrames for everything; Datasets are the typed JVM sibling.
Practice5 prompts
Why is there no separate Dataset API in PySpark?
Which abstractions get Catalyst optimization, and which does not?
True or false: an RDD is type-safe. Explain what kind of typing an RDD[Person] actually has.
When does a misspelled column name fail in a DataFrame — compile time, plan-analysis time, or mid-job?
Why is a PySpark DataFrame not meaningfully slower than a Scala DataFrame, and when does a Python performance gap actually appear?
05 · Edge cases
Edge Cases: Nulls, Schemas & Surprises
NULL is "unknown", not "zero" and not "false". Any operation touching an unknown produces an unknown, and filters silently discard unknowns. So you must decide what unknown means before you compute on it.
⏱ 6 min · Topic 5 of 7
The structured API is happy-path-friendly, which is exactly why its failure modes are sneaky: they rarely throw. Instead they silently drop rows, turn numbers into nulls, or corrupt your data on read. This section is the set of surprises that bite real pipelines — each shown as a small input that produces a result you did not expect.
The unifying cause for most null surprises is that Spark SQL uses three-valued logic: a comparison or arithmetic involving NULL evaluates to NULL (which is neither TRUE nor FALSE). filter keeps only rows where the predicate is TRUE — so NULL predicates are dropped. Knowing this one rule explains most of what follows.
Core mental model
NULL is "unknown", not "zero" and not "false". Any operation touching an unknown produces an unknown, and filters silently discard unknowns. So you must decide what unknown means before you compute on it.
Why it matters
These are the bugs that pass code review, pass a small test, and then quietly under-report a metric or lose records in production — the worst kind, because nothing errors. Senior engineers reach for isNotNull and coalesce reflexively and never trust schema inference on raw text.
three-valued logic
SQL boolean logic with TRUE, FALSE, and NULL (unknown); comparisons involving NULL return NULL, and filter keeps only TRUE.
coalesce()
Returns the first non-null argument; the standard way to supply a default for a possibly-null column before computing on it.
isNotNull() / isNull()
Column predicates that test for presence/absence of a value; used to handle nulls explicitly in filters.
nullable (metadata)
The StructField flag describing whether a column may be null. It is optimizer metadata only — Spark does not enforce it.
A filter silently drops null-age rowsworked example
Bo is correctly excluded (17 is not > 18). But Cy disappears too: NULL > 18 evaluates to NULL, filter keeps only TRUE, so the null-age row is dropped with no warning. The fixed version makes the choice explicit — and if you wanted to keep unknown-age rows you would write col("age").isNull() | (col("age") > 18).
Three-valued logic in action. The bug is invisible: no error, no log line — just a quietly smaller result. Always handle nulls explicitly in predicates.
NULL propagates through arithmetic (it is not zero)worked example
col("price") * 0.9 is NULL for A2 because any arithmetic on NULL is NULL. coalesce(col("price"), lit(0)) substitutes a default before the multiply, so the fixed column is 0. Whether 0 is the right default is a business decision — the point is that NULL forces you to make it.
NULL is not 0. coalesce(col, lit(default)) is the idiom for "use this value when the column is missing" before you compute on it.
inferSchema corrupts leading-zero ids on CSV readworked example
inferSchema read "001" as the integer 1 — the leading zeros are gone and the ids no longer match anything downstream. This is silent corruption. An explicit string schema keeps them intact. inferSchema also triggers an extra full pass over the file just to guess types, and is prone to date/timestamp ambiguity. This is CSV/JSON-specific: Parquet carries its own schema, so it does not infer — see the Reading & Writing Data chapter for format detail.
The strongest argument for explicit schemas: inference is a guess made from a sample, and a wrong guess corrupts data with no error. Never use inferSchema on production text data.
Common mistake
Filtering with col("x") > n without considering nulls. Null-valued rows are silently dropped (NULL predicate is not TRUE), quietly shrinking your result with no error.
Trusting inferSchema on raw CSV/JSON in a pipeline. Leading zeros are stripped, dates are mis-parsed, and an extra full read pass is paid — all silently. Declare an explicit schema instead.
Better habit
Make null handling explicit: isNotNull() in predicates, coalesce() before arithmetic.
Always read production text data with an explicit schema; never inferSchema.
Leave nullable=True unless you have truly guaranteed non-null data.
nullable=False is a promise Spark does not enforce
nullable=True is the default. The nullable flag is optimizer metadata, not a constraint: Spark does NOT validate it at write time. If you declare a column nullable=False but the data actually contains nulls, the optimizer may make assumptions (e.g. skip null checks) that produce wrong results rather than an error. Only set nullable=False when you have already guaranteed non-null data.
Ambiguous column after a join
Join two DataFrames that both have an "id" column and then select("id"), and Spark raises AnalysisException: Reference 'id' is ambiguous. Disambiguate or rename the column before selecting. The mechanics of joins and how to resolve this live in the Joins in Spark chapter (Chapter 11) — this is just the heads-up.
Remember this
NULL is "unknown": it silently drops rows in filters and poisons arithmetic, so handle it explicitly with isNotNull()/coalesce(). Never trust inferSchema on raw text — declare the schema. nullable is metadata, not a constraint.
Practice4 prompts
Given a column age with some nulls, write a filter that keeps adults AND keeps unknown-age rows.
Why does col("price") * 0.9 produce NULL for a missing price, and how do you make it 0 instead?
Explain how inferSchema can corrupt an order_id column of 001/002/010, and the one-line fix.
Is nullable=False enforced by Spark? What can go wrong if you declare it on data that has nulls?
06 · Summary
Recap & What’s Next
DataFrame = rows + schema, distributed + lazy. The schema makes the plan; the plan makes optimisation; built-ins keep you in the plan; nulls and inference are the silent edges that bypass it.
⏱ 4 min · Topic 6 of 7
A DataFrame is a distributed, lazy table with a schema: named, typed columns whose rows are split into partitions across the cluster. You build logic from Column expressions — col(), df["x"], expr(), withColumn, select — and you keep Spark in its fast path by composing built-in functions (Catalyst-visible) rather than Python UDFs (opaque). The schema is the linchpin: it is what lets Spark turn your code into an optimised physical plan you can inspect with explain().
You also saw the triad precisely. RDD = untyped, no schema, no optimizer. DataFrame = schema + Catalyst, checked at plan-analysis time, the same speed in Python as in Scala. Dataset = compile-time typed, Scala/Java only; a DataFrame is conceptually a Dataset[Row]. And you saw the edges: NULL three-valued logic silently dropping rows, NULL poisoning arithmetic, nullable being metadata not a constraint, and inferSchema silently corrupting data.
The table below consolidates the triad for quick recall, then there are the three mistakes that cause the most damage and a hands-on task to make it stick.
Core mental model
DataFrame = rows + schema, distributed + lazy. The schema makes the plan; the plan makes optimisation; built-ins keep you in the plan; nulls and inference are the silent edges that bypass it.
Why it matters
Everything from Chapter 7 onward — reading sources, joins, aggregations, window functions, optimization — is written in this structured API. The schema model and the null/inference edges you learned here are the foundation those chapters assume you already have.
plan-analysis time
When Spark validates column names and types against the schema (an AnalysisException fires here) — before execution, after compile.
explain()
Prints the physical plan Spark chose for a DataFrame; your window into what the optimizer did.
RDD vs DataFrame vs Dataset — the consolidation table
Dimension
RDD
DataFrame
Dataset
Schema
None
Named, typed columns
Named, typed columns
Optimizer visibility
None (opaque to Catalyst)
Full (Catalyst + Tungsten)
Full (Catalyst + Tungsten)
Type safety
None (no structural checking)
Plan-analysis time (runtime, pre-execution)
Compile-time (typed JVM objects)
Language availability
All
All
Scala / Java only
When to use
Rare: unstructured data, custom partitioning
Default for everything (the only structured API in PySpark)
Typed Scala/Java domain logic
Common mistake
Reaching for a Python UDF when a built-in function exists. The UDF is opaque to Catalyst and runs row-by-row across the JVM-Python boundary; you lose optimisation and pay serialization cost. (Chapter 14.)
Filtering or computing without handling nulls. NULL three-valued logic silently drops rows and poisons arithmetic — wrong results with no error.
Using inferSchema on production text data. Silent corruption (leading-zero ids become integers, dates mis-parse) plus an extra full read pass. Always declare an explicit schema.
Better habit
Default to DataFrames; know that Datasets are the typed Scala/Java sibling and RDDs the untyped foundation.
Build logic from col() + built-in functions; reach for a UDF only when no built-in exists.
Declare explicit schemas, handle nulls explicitly, and run explain() when a result looks wrong.
The whole chapter in one sentence
A DataFrame is a distributed, lazy, schema-bearing table you manipulate with column expressions; the schema is what lets Catalyst optimise your code — and nulls plus schema inference are the silent edges that quietly bypass that safety.
20–30-minute hands-on task
Try the hands-on task in the practice prompts below: build the people DataFrame, write a null-safe filter, apply coalesce, read a CSV with and without inferSchema, and call explain().
What interviewers test from this chapter
"DataFrame vs Dataset vs RDD?" Strong answer: RDD = no schema/optimizer and no type safety (an RDD[Person] only has host-language generic typing); DataFrame = schema + Catalyst, validated at plan-analysis time, a Dataset[Row] under the hood, and the same speed in Python as Scala because execution is on the JVM; Dataset = compile-time typed, Scala/Java only. Bonus points for naming a null/inference pitfall and for noting the Python gap only appears with UDFs/RDD lambdas.
Remember this
DataFrame = distributed, lazy, schema-bearing table; built-ins keep Catalyst optimising; RDD/DataFrame/Dataset differ on schema, optimizer, and type safety (and PySpark only has DataFrames); nulls and inferSchema are the silent edges to guard.
Practice8 prompts
From memory, fill the RDD/DataFrame/Dataset table across schema, optimizer, type safety, and language.
Name the three highest-damage mistakes from this chapter and the one-line fix for each.
Which chapter owns Catalyst, which owns UDFs, and which owns joins?
Hands-on (step 1 of 6): Start a local Spark session, build the people DataFrame with an explicit StructType, and add one row with a null age.
Hands-on (step 2–3 of 6): Run people.filter(col("age") > 18).show() and count how many rows disappeared vs the input — confirm the null row vanished. Then rewrite the filter using isNotNull() so no rows are silently lost.
Hands-on (step 4 of 6): Add a price column with one null and run withColumn("disc", col("price") * 0.9); observe the NULL result, then fix it with coalesce(col("price"), lit(0)).
Hands-on (step 5 of 6): Write the data to a small CSV with an order_id of "001", read it back with inferSchema=True and confirm the id became 1; re-read with an explicit string schema and confirm it stays "001".
Hands-on (step 6 of 6): Call people.select("name").explain() and locate the Project and Scan nodes in the physical plan printout.
07 · Next Chapter
Next Chapter
You can now create DataFrames, declare schemas, and build column expressions, the structured API in your hands.
⏱ 3 min · Topic 7 of 7
Next chapter
Reading & Writing Data: Sources & File Formats
You can now create DataFrames, declare schemas, and build column expressions, the structured API in your hands.
Next, Chapter 7 connects Spark to the outside world: reading and writing CSV, JSON, Parquet, ORC, and Avro; schema inference vs explicit schemas; partitioned reads and writes; and save modes.
