APACHE AIRFLOWAuthoring DAGs

Connections, Variables & Secrets Management

How data engineers make pipelines run themselves — pick a topic on the left and its full breakdown loads here: the mental model, runnable DAGs, the failure modes that wake people at 3am, and the judgment that separates a scheduled script from an orchestrated system.

⏱ 18 min readTopics chapter readerLevel · Medium
01 · Orientation

What You’ll Master Here

Your DAG names things it needs; the environment supplies them. The code says conn_id="warehouse" and never knows what that resolves to.

⏱ 4 min · Topic 1 of 8

Connections hold credentials. Variables hold configuration. Both are resolved at run time from outside your DAG file, which is what lets the same code run against a laptop database in development and a warehouse in production without a single line changing.

This chapter covers the model, the exact lookup order Airflow uses, and the one performance mistake that appears in almost every Airflow deployment eventually.

Core mental model

Your DAG names things it needs; the environment supplies them. The code says conn_id="warehouse" and never knows what that resolves to.

Why it matters

Getting this right is what makes environment promotion safe and what keeps credentials out of version control. Getting it wrong produces either a secret in a git history — which is a permanent problem — or a scheduler degraded by configuration lookups.

Connection
A stored credential and endpoint, retrieved by conn_id. Never appears in your DAG file as a value.
Variable
A stored configuration value, retrieved by key. Suitable for settings, never for secrets you can put in a connection instead.
conn_id
The identifier your code passes. The only part of a connection that belongs in version control.
One connection, three ways to define it
Connection type
Host
1. As an environment variable — best for local dev and CI
export AIRFLOW_CONN_WAREHOUSE='postgres://airflow:***@warehouse.internal:5432/analytics'
2. In a secrets backend — best for production
# AWS Secrets Manager, under the configured prefix
airflow/connections/warehouse  ->  postgres://airflow:***@warehouse.internal:5432/analytics
3. In your DAG — always by id, never by value
hook = PostgresHook(conn_id="warehouse")
# the credential is resolved at run time, on the worker
The DAG never changes between environmentsOnly the conn_id appears in your code. Dev points it at localhost, production points it at the warehouse, and the DAG file is byte-identical — which is what makes promotion between environments safe.
Common mistake

Putting a credential directly in a DAG file, even temporarily. It enters git history permanently. Removing the line does not remove the secret, and rotating it becomes mandatory rather than optional.

Better habit

Reference conn_id and nothing else. If a credential appears in a diff, stop.

Use identical conn_ids across environments so promotion changes no code.

Prefer connections over variables for anything credential-shaped — connections are built for it.

Name it, do not hold it

Your DAG names the connection it needs. Resolution happens on the worker, at run time, from whichever source the deployment is configured with.

Remember this

The DAG names what it needs; the environment supplies it. That separation is what makes one file work everywhere.