What You'll Master Here
format, options, schema, and path. Choose columnar formats, supply explicit schemas, and partition smartly — the rest of your job gets faster for free.
Spark is a compute engine, so the first and last thing every job does is read and write data. This chapter makes you fluent in the read/write API and, more importantly, in the choices that govern your job's speed and cost: file format, schema handling, partitioning, and save mode.
Those choices compound. Storing the same dataset as Parquet instead of CSV lets Spark read only the columns it needs and skip entire row groups that cannot match your filter — the I/O savings come from three layered mechanisms (column pruning, predicate pushdown, and data skipping) rather than any single trick. Databricks reports a ~34x query speedup in one benchmark converting 1 TB of CSV to Parquet; the real figure depends on schema, selectivity, and compression — the mechanism is columnar reads + pushdown + compression.
You will leave with runnable read/write code, a clear picture of when each format wins, defences against schema inference and dirty records, safe partition and save-mode patterns, and active-recall exercises to lock it all in.
Reading and writing is a four-part recipe: format, options, schema, and path. Choose columnar formats, supply explicit schemas, and partition smartly — the rest of your job gets faster for free.
Format and partitioning choices are among the highest-leverage decisions in a data platform. Picking Parquet over CSV and partitioning sensibly can cut runtime and cloud bills dramatically with no change to your logic.
- DataFrameReader / Writer
- spark.read… and df.write…, the entry points for I/O.
- columnar format
- Stores each column together (Parquet, ORC); great for analytics.
- partitioned table
- Data physically split into folders by a column, enabling partition pruning.
- save mode
- How a write behaves if the target exists: append, overwrite, errorifexists (default), ignore.
Defaulting to CSV/JSON for large analytical data. Every query reads every byte; converting to Parquet first usually pays for itself on the very first re-read.
Prefer Parquet for analytical data; use CSV/JSON for interchange only.
Supply explicit schemas for production reads.
Partition large tables by a low-cardinality column you filter on.
The same data, stored well (columnar + partitioned), is a fraction of the cost to query. Format and layout are performance decisions, not afterthoughts.
I/O is read/write with a format, options, schema, and path. Columnar formats (Parquet/ORC) and sensible partitioning make every downstream query faster and cheaper.
- When would you choose CSV over Parquet, and when not?
- Name the three layered mechanisms that make Parquet faster than CSV for a selective analytical query.