Status
Motivation
Airflow uses synchronous programming in all of its components except the Triggerer, which manages and executes triggers asynchronously. Airflow has many blocking I/O tasks in the other components, particularly SQLAlchemy queries used in the scheduler, API, webserver, and executor, as well as methods used to execute and monitor Airflow tasks in external systems by remote executors. By migrating these methods and queries to asynchronous programming, Airflow will be much faster and consume fewer resources than ever before.
Considerations
One of the requirements of this migration is to integrate Asynchronous SQLAlchemy into the Airflow core and provide an asynchronous session for use in Asyncio coroutines. This integration can reduce the execution time of processing based on multiple independent queries and enable Airflow to work more easily concurrently with low resource consumption.
What change do you propose to make?
This AIP proposes:
- adding support for asynchronous SQLAlchemy in Airflow core by inferring asynchronous connection (SQLAlchemy engine) from metadata connection configuration
- implementing an asynchronous version of the secrets backend to retrieve connections and variables asynchronously
- migrating the API and webserver to asynchronous programming
- migrating the scheduler to asynchronous programming
- migrating the executors to asynchronous programming