Connections, Variables & Secrets Management

Tasks need logins and settings. Airflow keeps them outside your DAG file, in two kinds of record:

Your DAG only names them, such as conn_id="shop_db". Each environment fills in its own values. So the same code runs on your laptop and in production.

A connection is a small record: a type, a host, a login, a password, a port, a schema and extras. You can write it as one URI:

postgres://shop_user:p%40ss%2Fword@db.example.com:5432/shop

Without that encoding, the first @ ends the login part. From postgres://shop_user:p@ss/word@db.example.com:5432/shop, Airflow read password p and host ss.

The JSON form needs no encoding, so it is safer to write by hand. Set it in an environment variable named AIRFLOW_CONN_ plus the conn_id in upper case:

export AIRFLOW_CONN_SHOP_DB='{"conn_type": "postgres", "host": "db.example.com",
  "login": "shop_user", "password": "p@ss/word", "port": 5432, "schema": "shop"}'

A connection can live in three places. Airflow searches them in this order, and stops at the first match:

1. A secrets backend, such as AWS Secrets Manager or HashiCorp Vault, if you set one up.
2. Environment variables, such as AIRFLOW_CONN_SHOP_DB.
3. The metastore: the Airflow database, which the UI and CLI write to.

We put shop_db in two places with different hosts. The task read the environment variable:

That leads to a common surprise. You edit a connection in the UI, and the UI saves it to the metastore. But a copy in an environment variable or a backend is found first, so tasks keep the old value.

So keep each connection in one place per environment. If a change seems to have no effect, look for a copy higher in the list.

A Variable is read with Variable.get. Where you call it matters, because Airflow parses the DAG file again and again, by default every 30 seconds (chapter 3):

# Wrong: runs on every parse, all day
CONFIG = Variable.get("shop_config", deserialize_json=True)

# Right: runs only when the task runs
@task
def load_tables():
    config = Variable.get("shop_config", deserialize_json=True)
    print(f"tables={config['tables']} batch={config['batch_size']}")

# Also right: a template, filled in when the task runs
BashOperator(task_id="list_tables",
             bash_command="echo {{ var.json.shop_config.tables }}")

A read at the top of the file asks the Airflow database on every parse, even when no task runs. Many files doing this slow down the scheduler for every DAG, not only for yours.

When scheduling gets slow, search your DAG files for Variable.get outside a function. It is a quick check.

Settings that change together belong in one JSON Variable. shop_config holds the tables and the batch size, so one read gets both.

Six separate Variables mean six reads. They also mean six values to update when you move to a new environment, and it is easy to forget two.

Choose the right home for each value:

Airflow hides secrets in task logs and shows *** instead. It hides connection passwords, and Variables whose names contain words such as password, secret, token or api_key.

It hides the exact secret text, even inside a longer string. But it cannot spot a changed form of the secret:

print(f"password={c.password}")
print(f"dsn=postgres://{c.login}:{c.password}@{c.host}/shop")
print("b64=" + base64.b64encode(c.password.encode()).decode())
print("api_key=" + Variable.get("api_key"))
print("region=" + Variable.get("region"))

So never print secrets, in any form. Task logs are often readable by many more people than the secret store.

In production, keep connections in a secrets backend. It gives you access rules, an audit log and password rotation in one place.

After a rotation, a task that already read the connection keeps the old password. The next task reads the new one. So rotate when few tasks run, or keep the old password valid for a short time.

Name it in the code, keep it outside:

1. Connections hold logins. Variables hold settings. DAGs only name them.
2. In a URI, encode @ and /. Or use the JSON form.
3. Lookup order: secrets backend, then environment variables, then the metastore.
4. Read Variables inside tasks or templates, never at the top of the file.
5. Airflow hides exact secrets in logs, but not changed forms. Never print them.

Check yourself: you changed shop_db in the UI, and tasks still use the old host. Why? (Answer: a copy in an environment variable or a backend is found first.)