APACHE AIRFLOWScale & Production

Assets & Cross-DAG Data-Aware Scheduling

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

An asset is a name for a piece of data. Producers announce they updated it; consumers subscribe to it. Neither needs to know the other exists.

⏱ 4 min · Topic 1 of 7

Chapter 1 argued that time is a bad proxy for readiness, and then spent fourteen chapters applying that inside a single DAG. This chapter applies it between DAGs.

Instead of scheduling a downstream pipeline for “probably after the upstream one finishes”, you declare what it consumes and let Airflow run it when that data actually appears. In Airflow 3 the object is called an Asset — it was called a Dataset until 3.0.

Renamed in Airflow 3.0 — the concept is the same, the imports are not
from airflow.sdk import Asset

orders = Asset("s3://lake/orders/")

# Producer
@dag(schedule="@daily")
def ingest():
    @task(outlets=[orders])
    def write_orders(): ...

# Consumer — runs when the asset is updated
@dag(schedule=[orders])
def report(): ...

# Or the shorthand for "one task produces one asset":
@asset(uri="s3://lake/orders/", schedule="@daily")
def orders(): ...
The current name, plus an @asset decorator that did not exist beforeEverything you knew about datasets carries over — the mechanism is unchanged. What changed is the name, the import path, and the addition of an @asset decorator for the common case where one task produces one asset.
Core mental model

An asset is a name for a piece of data. Producers announce they updated it; consumers subscribe to it. Neither needs to know the other exists.

Why it matters

Cross-DAG timing is where the Chapter 1 problem reappears at a larger scale. A report scheduled thirty minutes after the ingest DAG is the same guess as a crontab, with the same failure on the same bad nights — and now it spans team boundaries, so nobody owns the mismatch.

Asset
A logical grouping of data identified by a URI. Renamed from Dataset in Airflow 3.0.
outlets
What a task declares it produces. Succeeding emits an asset event.
Asset event
The record that an asset was updated. What the scheduler reacts to when deciding whether a consumer should run.
Common mistake

Following a tutorial that uses Dataset on an Airflow 3 deployment. The import path changed to airflow.sdk.Asset. The mechanism is identical, but the code will not run — and the tutorial’s age tells you something about the rest of its advice.

Better habit

Use a URI that describes where the data actually lives, so the name means something to a human reading lineage.

Declare outlets on the task that genuinely produces the data, not on a downstream notification task.

Check which Airflow version any dataset or asset material targets before applying it.

Chapter 1, at DAG scale

Scheduling a consumer for “thirty minutes after the producer” is a crontab. An asset subscription is the dependency, expressed properly.

Remember this

Assets let a downstream DAG wait on data rather than on a clock. Same idea as Chapter 1, one level up.