D8LooP
Start building

Hands on Concepts

Real-Time and Streaming Data Engineering

Four things a batch engineer believes that stop being true in a stream.

4 shifts35 steps~8h total
Start this plan

It opens on your dashboard and ticks itself off as you go. Every step below is a chapter or an exercise that already ships.

01

When you finish, you can

You can run a stream in production and explain what it guarantees.

Stream owned You can run a stream in production

02

The plan, 4 shifts

Ordered by what each one buys you, not by topic. Every step says why it is here.

  1. Shift 01 · ~2h

    Time stops being one thing

    You reason in event time and know when a window is allowed to close.

    Skip this if you already store event time and ingest time separately.

    1. SectionEvent time vs ingestion time6 minTwo clocks. A pipeline storing one of them can never be corrected.
    2. SectionWindowing an endless stream7 minYou cannot aggregate the infinite. Windows are how you make it finite.
    3. SectionEvent time, processing time & watermarks7 minThe watermark is a promise about lateness. Say the number.
    4. SectionEvent time & windows7 minThe same idea, with a real API and state store underneath.
    5. InterviewWindow types, and what closes them6 minWindows group. The watermark decides when a group is finished.
    6. ExerciseAssign events to event-time windows40 minin the studioPut events in windows by hand once. The boundaries stop being abstract.
    7. ExerciseInterview late-data replay check18 minin the studioLate arrivals against a window you already published. Now what?
    8. QuestionsData Pipeline · Conceptual15 minWatermarks, lateness and completeness, said precisely.

  2. Shift 02 · ~2h

    Delivery stops being once

    You can name your delivery guarantee and what it costs downstream.

    Skip this if you already treat every consumer as at-least-once.

    1. SectionThe log: producers, broker, consumers6 minA stream is an append-only log with a cursor. Everything follows.
    2. SectionPartitions: scale and ordering6 minParallelism and ordering are one setting. You trade them.
    3. InterviewWhat ordering a stream gives you6 minPer partition, never global. A flat "yes" designs a reordering bug.
    4. SectionDelivery semantics in a stream7 minAt-least-once is what you get. The rest you build.
    5. SectionExactly-once: checkpoints & transactions7 minExactly-once is a property of the write, not of the broker.
    6. SectionPoison messages & dead-letter queues6 minOne bad record can stall a partition forever. Park it.
    7. InterviewWhere the dead-letter queue goes6 minThere is more than one, and the stream itself is not one.
    8. ExerciseMeta: deduplicating an at-least-once stream35 minin the studioDeduplicate an at-least-once stream. The canonical version of this.
    9. QuestionsData Pipeline · Failure & recovery15 minReplays and redeliveries, with the target state after each one.

  3. Shift 03 · ~2h

    State stops being free

    You can bound the memory a never-ending job is allowed to hold.

    Skip this if you can name what grows in every streaming job you run.

    1. SectionThe unbounded table model7 minA stream as a table that keeps growing. The useful abstraction.
    2. SectionOutput modes, triggers & execution6 minAppend, update, complete. Choosing wrong rewrites the world each batch.
    3. SectionStateful streaming patterns6 minThe same problem without a framework. It clarifies what a framework does.
    4. SectionMemory safety & backpressure6 minName the thing that grows. Then bound it, or expire it.
    5. SectionCheckpointing: cutting long lineage6 minA stream with no checkpoint restarts from nothing, or from wrong.
    6. InterviewDistinct users in a rolling window6 minSketches merge and never subtract. That decides your precomputation.
    7. ExerciseDeduplicate a stream in bounded memory40 minin the studioDedupe a stream you cannot hold. The bounded-memory classic.
    8. ExerciseAlert with hysteresis, not on every sample45 minin the studioAlert with hysteresis. State that has to be right, not just small.
    9. QuestionsApache Spark · Conceptual & architecture15 minThe runtime underneath the stream. Say what the driver is holding.

  4. Shift 04 · ~2h

    Failure stops being a rerun

    You can correct a stream that has already published wrong numbers.

    Nothing to skip — replay decides whether you can run this.

    1. SectionKappa: one streaming path6 minOne path, corrected by replay. It only works if replay actually works.
    2. SectionChoosing an architecture6 minTwo code paths that must agree, or one that must be replayable.
    3. SectionLate-arriving data & replays7 minThe number you published was right then. It is not now.
    4. SectionWhat changing your mind actually costs6 minPrice the reversal before you commit to never exiting.
    5. InterviewBatch to streaming, safely6 minParallel run, compare, cut over. The batch path is your baseline.
    6. InterviewOn-demand or provisioned capacity6 minA stream bills continuously. Predictability is worth real money.
    7. ExerciseRealtime events pipeline45 minin the studioDesign one end to end and defend the guarantee you claimed.
    8. ExerciseYou fixed it. Who already used it?50 minin the studioYou fixed it and replayed. Who already used the wrong numbers?
    9. QuestionsData Pipeline · Trade-offs & tool choice15 minKafka or a nightly extract. Say what would change your mind.

03

Start it now

Reading is free and needs no account. The plan itself tracks on your dashboard.

Start Real-time & streamingTracked, 4 shifts, ~8hSee the other plans16 more, by what you are aiming at