PRODUCTION FLOWSExecution Models

Batch vs Streaming vs Micro-Batch

How data moves in production — pick a topic on the left and its full breakdown loads here: the mental model, real pipeline diagrams and worked examples, failure modes, and the habits that keep data flowing correctly and on time.

⏱ 18 min readTopics chapter readerLevel · Foundations
01 · Orientation

What You'll Master Here

A triggered run is a relay race — each leg carries the batch to the line, hands off, and stops. A continuous run is a conveyor belt that never stops. Micro-batch runs that same relay every few seconds; it never becomes the belt.

⏱ 5 min · Topic 1 of 11

Chapter 2, The Data Pipeline Lifecycle, separated boundedness, a source property, from execution, how you run over it. This chapter is the execution half. Batch and streaming are two execution models, not two speeds. The widget below sorts eight such statements into Triggered or Continuous execution.

Triggered execution plans a job over a finite slice, runs it to completion, commits, and exits. Continuous execution never exits — records flow through one at a time. Micro-batch is triggered execution with the slice made small, tunable from a day down to about a second. That it is "a bit of streaming" is backwards — it is batch, run in a loop, which is why Spark 4.1 needed a new engine, Real-Time Mode.

Core mental model

A triggered run is a relay race — each leg carries the batch to the line, hands off, and stops. A continuous run is a conveyor belt that never stops. Micro-batch runs that same relay every few seconds; it never becomes the belt.

Why it matters

Confusing these for a speed dial causes two costly mistakes: expecting a faster trigger to become streaming, and reaching for a continuous engine when a smaller slice would do.

triggered execution
An execution model where the engine plans a job over a finite slice, runs it to completion, commits, and exits.
streaming
An execution model of long-lived tasks that process records as they arrive, with no batch boundary — defined by architecture, not speed.
Northwind Energy — 500,000 smart meters, one execution questionSort each statement into Triggered execution or Continuous execution.6 still in the tray
Tray — click a card's destination to place it
  • Commits its offsets, then the tasks exit
  • Keeps its state in the task's own memory, with no batch boundary at which the task ever exits
  • A nightly job and a 30-second trigger are the same machinery
  • Spark Structured Streaming with trigger(processingTime = "30 seconds")
  • You can shrink the slice from 24 hours down to about a second
  • Below about a second, you are not shrinking a slice — you are changing engines
Triggered execution1 placed, 1 correct
  • Plans a job over a finite slice of input✓ correct home
Continuous execution1 placed, 1 correct
  • Tasks never terminate✓ correct home
Common mistake

Treating micro-batch as "a bit of streaming" instead of batch with a smaller slice. You expect shrinking the trigger to reach streaming and are surprised Spark needed a new engine.

Better habit

Ask whether a run exits or never exits before calling it batch or streaming.

Treat micro-batch as tunable batch, not cheap streaming.

Default to triggered execution; earn a continuous engine with a stateless, millisecond-critical decision.

The big idea

Batch and micro-batch share one triggered model, tunable to about a second. Continuous is different — why Spark built a new engine, not a lower number.

Remember this

Batch and streaming are two execution models, not two speeds; micro-batch is triggered execution with the slice turned down, short of continuous.

Practice2 prompts
  1. A parking garage recomputes occupancy every 15 minutes. Triggered or continuous execution, and why?
  2. Name one thing, not a number, that must change to turn a 10-second micro-batch job into a continuous one.