Apache Spark

The big-data problem, MapReduce's limits, in-memory computing, and the unified engine for batch, SQL, streaming, and ML.

The runtime: driver and executors, jobs → stages → tasks, cluster managers (standalone, YARN, Kubernetes), and client vs cluster mode.

SparkSession, running code in notebooks or via spark-submit, and reading the Spark UI end to end — jobs, stages, tasks, executors, storage, and what a failed job looks like.

Resilient Distributed Datasets: partitions, lineage, fault tolerance, and when low-level RDDs still matter under the modern APIs.

Narrow vs wide transformations, lazy evaluation, and how an action triggers a DAG that becomes stages and tasks.

The structured APIs: DataFrames and Datasets vs RDDs, schemas, columns, and typed vs untyped processing.

Data sources and formats (CSV, JSON, Parquet, ORC, Avro), schema inference vs explicit schemas, partitioned reads/writes, and save modes.

The everyday API: select/selectExpr and filter for rows and columns, withColumn/lit/cast/when to shape them, and sort/distinct/limit — plus the null and dedupe traps, chained into a readable, performant pipeline.

Strings, dates and timestamps, nulls, and complex types (arrays, maps, structs), plus regex and type casting.

groupBy and agg, multiple aggregations, rollup and cube, and pivot tables at scale.

Join types (inner, outer, semi, anti) and the join strategies Spark picks: sort-merge, shuffle-hash, and broadcast.

Ranking, lead/lag, and running aggregates over partitions, with window frame specifications.

SQL over DataFrames and temp views, and how Catalyst turns a query into a plan: parse → analyze → optimize → physical, with pushdown and pruning.

User-defined functions, pandas/Arrow UDFs, the Pandas API on Spark, and why to prefer built-ins over UDFs.

The execution lifecycle end to end: application, jobs, stages, and tasks; the DAG scheduler and shuffle boundaries.

Partitions as the unit of parallelism: repartition vs coalesce, partition sizing, and partition pruning.

What triggers a shuffle, how shuffle read/write works, where it spills, and the strategies that reduce or avoid it.

Detecting skew in the Spark UI and fixing straggler tasks with salting, isolate-and-broadcast, and AQE skew joins.

cache and persist, storage levels, when caching helps or hurts, and truncating long lineage with checkpointing.

Broadcast joins for dimension lookups, broadcast variables and accumulators, and bucketing to avoid shuffles on writes.

The executor memory model (execution vs storage), off-heap memory, Tungsten code generation, and diagnosing OOM and spills.

Runtime optimization with AQE (coalesce partitions, skew joins, join switching) and cost-based optimization from table statistics.

A repeatable tuning method: reading the Spark UI, the configs that matter, finding bottlenecks, and fixing the small-files problem.

The configs that actually matter and how to size them: executor cores/memory/overhead, partitions, AQE, and worked example configs for 10 GB, 500 GB, and 10 TB jobs, with the formulas to derive your own.

Streaming as an unbounded table: sources and sinks, output modes, triggers, and micro-batch vs continuous processing.

Event vs processing time, tumbling and sliding windows, watermarks for late data, stateful operations, and exactly-once.

Bringing ACID to the lake: transactions, time travel, schema enforcement and evolution, MERGE/upsert, OPTIMIZE/Z-order, and Auto Loader.

Submitting and sizing apps: client vs cluster mode, executor/core/memory config, dynamic allocation, and YARN vs Kubernetes.

Operating Spark for real: idempotent writes, retries and recovery, monitoring, cost tuning, and the anti-patterns to avoid.

A worked end-to-end job plus the interview skill of narrating it: from requirements to DAG, and optimizing a slow job step by step.