XComs, Templating & Jinja

You write a DAG file once, and it runs every day. But each run needs its own values, such as which day to read.

Those values come from two places:

This chapter shows both, and the mistakes that make a run look green while it does the wrong thing.

Each run gets a set of values called the template context. These are the ones you will use most:

You can read them in two ways. In a template field, write Jinja such as {{ data_interval_start }}. In a @task function, add a parameter with the same name:

@task
def load_orders(data_interval_start=None, data_interval_end=None):
    print(f"reading orders from {data_interval_start} to {data_interval_end}")

prev_data_interval_end_success helps with incremental loads. Suppose the run about 01-20 fails.

The next run then sees where the last good run stopped: 2026-01-20 00:00. So it can read from there, and no day is lost.

Chapter 5 showed a scheduled run. Its logical date equals the start of its interval. A manual run is different: someone pressed Trigger, or ran airflow dags test.

Both runs below cover the same day of orders, 2026-01-20:

The interval is the same in both runs. But ds is not. A manual run takes the last full interval, and its logical date is the time it was triggered for.

So a query with order_date = ‘{{ ds }}’ reads order 105 in the scheduled run, but order 106 in the manual run. Neither run fails. One of them is wrong.

An XCom is a small value that one task leaves for other tasks in the same run. In TaskFlow, a return value becomes an XCom, and an argument reads it (chapter 7).

In classic code, you pull it yourself with ti.xcom_pull. Always name the task with task_ids. In a template, the pull looks like this:

push_path = ...   # a @task that returns "s3://shop/orders/2026-01-20/"

pull = BashOperator(
    task_id="pull_no_edge",
    bash_command="echo got {{ ti.xcom_pull(task_ids='push_path') }}",
)
# no push_path >> pull line

A pull in a template does not create an edge. Without push_path >> pull, the scheduler may start pull first. Sometimes the order is lucky, so the bug comes and goes.

So write the edge next to every templated pull. Or use TaskFlow, where passing the value makes the edge.

Every XCom is saved in the Airflow metadata database. The scheduler and the UI use that same database all the time. The Airflow docs say XComs are for small amounts of data.

So pass a reference to the data, not the data itself:

# Wrong: the whole table goes into the Airflow database
@task
def load_orders():
    return read_all_orders()              # thousands of rows

# Right: the table goes to storage, and only the path is passed
@task
def load_orders(ds=None):
    path = f"s3://shop/orders/{ds}/orders.parquet"
    write_orders(path)
    return path

Good XComs are paths, table names, row counts and dates. If a value grows with the size of your data, it does not belong in XCom.

Airflow can also keep large XComs in object storage, through XComObjectStorageBackend in the common.io provider. It is useful at scale. But passing a path is still simpler.

Airflow fills in Jinja just before a task runs. But it does this only for the template fields of the operator (chapter 6). Everything else stays as plain text.

In this test, the same {{ ds }} sits in two places:

def show(day):
    path = "s3://shop/orders/{{ ds }}/"
    print(f"in body: {path}")
    print(f"from op_kwargs: {day}")

PythonOperator(
    task_id="classic",
    python_callable=show,
    op_kwargs={"day": "{{ ds }}"},
)

There is no error. The task writes to a folder whose name contains braces, and succeeds.

In TaskFlow, the arguments of a @task call are rendered too. load(path="s3://shop/orders/{{ ds }}/") gave path: s3://shop/orders/2026-01-21/ in our run.

Small values, and rendered arguments:

1. Run values come from the template context, and from XComs.
2. Choose rows with data_interval_start and data_interval_end, not ds or the clock.
3. A templated xcom_pull needs its own >> edge.
4. Pass paths and counts in XCom, never whole tables.
5. Jinja renders only in template fields. Check the Rendered Template tab.

Check yourself: a manual run for 2026-01-21 reads order_date = ‘{{ ds }}’. Which day does it read, and which day is its interval? (Answer: 2026-01-21, and 2026-01-20.)