What You’ll Master Here
Operators do work. Sensors wait for work to become possible. Hooks are the connection layer both of them sit on. Almost every authoring decision is choosing between those three.
Chapter 3 introduced operators, sensors and hooks as three shapes that all become one node in the graph. This chapter is the working detail: the built-in operators you will meet everywhere, how to build and ship your own, why sensors have a completely different cost model, and the constructor rules that quietly break custom operators.
The centrepiece is a full walkthrough of writing a custom operator — where the file goes, what the class looks like, how to make it templated, how to import it, and how to test it. That is the part most Airflow material skips.
Operators do work. Sensors wait for work to become possible. Hooks are the connection layer both of them sit on. Almost every authoring decision is choosing between those three.
The provider ecosystem is Airflow’s real moat — thousands of maintained integrations that already handle auth refresh, pagination and retry semantics. Engineers who do not know it exists rewrite that work badly. Engineers who over-trust it write a custom operator for three lines of Python.
- Provider package
- A separately versioned package of operators and hooks for one external system. Upgraded independently of Airflow itself.
- Hook
- The reusable client for an external system. Handles connection lookup and auth; operators are largely thin wrappers over hooks.
- Sensor
- A subclass of operator whose only job is to wait for an external condition to become true.
- BaseOperator
- The class every operator inherits from, imported in Airflow 3 as `from airflow.sdk import BaseOperator`.
| System | Provider package | Typical operator |
|---|---|---|
| AWS (S3, EMR, Glue) | apache-airflow-providers-amazon | S3KeySensor, EmrAddStepsOperator |
| Google Cloud | apache-airflow-providers-google | BigQueryInsertJobOperator, GCSToBigQueryOperator |
| Azure | apache-airflow-providers-microsoft-azure | WasbBlobSensor, AzureDataFactoryRunPipelineOperator |
| Snowflake | apache-airflow-providers-snowflake | SQLExecuteQueryOperator with a Snowflake conn |
| Postgres / MySQL | apache-airflow-providers-postgres | SQLExecuteQueryOperator, PostgresHook |
| Spark / Databricks | apache-airflow-providers-databricks | DatabricksSubmitRunOperator |
| Kubernetes | apache-airflow-providers-cncf-kubernetes | KubernetesPodOperator |
| dbt | apache-airflow-providers-dbt-cloud (or Cosmos) | DbtCloudRunJobOperator |
| Slack / email | apache-airflow-providers-slack | SlackWebhookOperator |
| Bash, Python, empty | apache-airflow (built in) | BashOperator, @task, EmptyOperator |
Writing a custom integration for a system that already has a provider. You reimplement auth refresh, pagination and retry semantics, and you maintain them forever. The provider version already handles the edge cases you have not met yet.
Search the provider list before writing any integration code. It takes a minute and usually ends the task.
Pin provider versions in requirements. They are versioned independently, so “it worked yesterday” is a real failure mode.
Treat waiting as a distinct kind of work with its own resource cost, not as a slow task.
Operators do, sensors wait, hooks connect. Hold that split and the rest of the chapter is detail.
Know what already exists before you write anything — then know exactly how to write the thing that does not.