D8LooPFocus mode

D8Loop · Field notes

Knowledge Corner.

Read the concepts, patterns, and interview signals behind every module — eight knowledge bases, read like an archive. Scroll to move through them; jump from the index on the left.

8 knowledge bases5 live now3 coming soon

Data architecture · 01

Data Modeling

Grain, keys, ownership, and governance decisions that make data products durable instead of merely diagrammed.

GrainKeysCardinalityNormalizationGovernanceHistory

Deep diveChoosing grain before naming tables PatternFacts, dimensions, and history without hand-waving Production noteData contracts and schema evolution in real teams

Open Data Modeling Knowledge Base

Production flows · 02

Data Pipeline

Ingestion, orchestration, lakehouse choices, streaming edges, and failure paths for systems that must be operated.

OrchestrationQuality gatesCloudLakehouseStreamingObservability

Deep diveDesigning a DAG that makes failure obvious PatternBatch, streaming, and lakehouse paths in one architecture Production noteRetries, idempotency, SLAs, and alert fatigue

Open Data Pipeline Knowledge Base

Distributed compute · 03

Apache Spark

Spark internals interviews actually probe: jobs, stages, and tasks; shuffles and skew; partitioning, caching, joins, and the memory model.

RDD vs DFShufflesSkewPartitioningBroadcastTuning

Deep diveHow a Spark job becomes stages and tasks PatternFixing data skew with salting and broadcast joins Production noteDiagnosing OOM, spills, and the small-files problem

Open Apache Spark Knowledge Base

Query systems · 04

SQL

Relational thinking, analytical query patterns, and the habit of explaining results before optimizing syntax.

JoinsAggregationsWindowsCTEsDebuggingPerformance

Deep diveHow interviewers read a query plan from your CTEs PatternWindow functions for retention, ranking, and deduping Production noteWhen a correct query becomes too expensive to ship

Open SQL Knowledge Base

Data transforms · 05

Python

Transformation code that stays legible under messy inputs, edge cases, and interviewer follow-up questions.

ParsingMungingValidationTestingContractsReadable ETL

{ }

Deep diveWriting transforms that reveal intent line by line PatternValidation before mutation in event normalization jobs Interview noteHow to narrate tradeoffs while coding live

Open Python Knowledge Base

Orchestration · 06

Apache Airflow

Scheduling semantics that trip everyone up, DAGs that survive review, and the concurrency, executor, and failure-mode decisions that separate a scheduled script from an orchestrated system.

DAGsSchedulingBackfillsExecutorsIdempotencyProduction

Deep diveHow Airflow schedules, retries, and backfills a DAG PatternWhy a daily DAG for Monday actually runs on Tuesday Production notePools, deferrable operators, and the sensor-storm problem

Open Apache Airflow Knowledge Base

Analytics engineering · 07

dbt

Models, refs, and materializations; tests, contracts, and lineage — the transformation layer most warehouses are actually built on, and the half of the stack Airflow triggers.

Models & refsMaterializationsSnapshotsTestsMacrosLineage

Deep diveIncremental models that stay correct at scale PatternLayering staging, intermediate, and mart models Production noteTests, freshness, and lineage that catch breakage early

Open dbt Knowledge Base

Question bank · 08

Interview Questions

A drilled bank of the questions data engineers actually get across SQL, Python, Spark, modeling, and design — with the reasoning each answer should show.

SQLPythonSparkModelingSystem designBehavioral

SQLWindow functions, top-N, and dedup under time pressure Sparkrepartition vs coalesce, OOM, and skew explained DesignDesigning a pipeline from a one-line prompt

Arriving soon

Build portfolio · 09

Data Engineering Projects

End-to-end builds that double as portfolio proof: ingestion, transformation, orchestration, and serving on realistic datasets and stacks.

IngestionELTOrchestrationStreamingLakehouseDashboards

StarterBatch ELT: API to warehouse with dbt and Airflow StreamingReal-time clickstream with Kafka and Spark CapstoneA lakehouse with quality gates and observability