LLM/Gen-AI mainly as the important trigger

Reiterating the fact that this needs more work, I do believe this can be incremental to Airflow. As Astronomer, we have worked on the LLM Providers which we contributed to Airflow late last year. But clearly, there is so more to do, both from building awareness of the patterns / templates to use, as well as patterns to support in Airflow to make these easier to use and adopt.

• Incrementally improve provider w/o breaking change to Airflow 3.0?

Need to submit DAGs in other ways than dropping them to a shared DAG folder

Different DAG distribution processes

• Provide an alternative to Git Sync and let the DAGs deployed by other means.

• Would it help to develop a better "dry run" for all major operators such that local development can test better based on mocks?

Ability to run tasks with workflow with "affinity" so that they can share inputs/outputs in shared CPU/GPU memory

• Allow templating to "executor" and "queue" fields such that by workflow logic tasks can be routed to a specific endpoint/runner

Ability to integrate seamlessly with other workflow engines - making Airflow a "workflow of workflows"

• Enhancements to provider packages with deeper integration (Github/Gitlab/ADO/which?)

Maintaining 700+ Python dependencies within all complexity of provider packages

As we have with the integrated provider packages a complex ecosystem we need to fix CI almost every second day. It would be good to think about modularization of provider packages and code - even with keeping a monorepo such that dependencies in provider packages can be modeled independent (e.g. keep a Venv per provider) and not a global Python site-package tree. This was very visible in migration to Python 3.12.

• Allow provider packages to be executed in a Venv such that not all python dependencies must get installed together in the same site-package tree. But on the other hand Operators need to be parseable in the DAG, so Operator parsing and execution might need to be un-bundled.

First user experience: Most common failures in DAG authoring

Many user fail in their first DAG editing (if they do not start with a copy&paste template) because with the Python code they do not understand the difference between the DAG parsing and execution stage. It is not directly visible what "global code" and code between operators/tasks is executed when. Compensation is the path to Jinja Templating which is also in most cases added complexity. TaskFlow approach had made it better in many cases but is not a viable approach for many operator which just consume "properties" as configuration.
If you compare users how they can easily define pipelines in Github or ADO in YAML, for many users a YAML based definition might be a much easier entry point.
I propose we re-discuss how DAGs are modelled such that "pitfalls" and "first user problems" are not needed to be answered via TaskFlow.

• For YAML maybe integrate something liek "DAG Factory" into the core as alternative if no real Python Code for DAGs is needed. Still leave the Pythonic approach of DAG creation to power users of course.
• Today the UI very much in the hidden secret detail pages shows the "Rendeed Template" fields. But for the users they need to manually assemble the "final" properties which are applied to the Operator at point of execution (together with the properties on another page that are not templated) - Would be cool to have a better view of actually which parameters alltogether (templated or not) were used to execute a task in the graph (Mainly for non TaskFlow tasks)

Airflow User Improvements

• AIP-63: DAG Versioning - the most requested feature in the Airflow User Survey
• AIP-38 Modern Web Application - also comes up a lot
• Different execution mechanisms

• Add capacity to UI shortcomings. UI team is anyway "under-staffed", it could well be that also Airflow 3.0 will suffer from this and raising a 3.0 discussion w/o capacity on UI will not make UI better than today.

Easy adoption of Airflow by new users

We have discussed this many times, but we absolutely need to make the individual first-time adoption of Airflow better.

I think the most common term I recall here is the notion of "Airflow Standalone", but whatever the term may be, an ultra quick, simple install of Airflow and the getting started experience is something we owe our community.

• Would it help to develop a better "dry run" for all major operators such that local development can test better based on mocks?

Integration improvements / Provider maintainability

The changes we made as part of Airflow 2.0 to split the Core Airflow releases from the Provider releases was clearly a good choice and made a huge impact. However, the integration maintainability balanced with growth still seems like it could use a significant set of improvements. Elad and I spoke about this a couple of days ago as well and I don't have a clear set of next steps here, but definitely worth exploring.

DAG / Workflow Support for non-linear complexity.

DAGs are only one-way. And findemantally this is because loops would call a task multiple times and you would need to keep the context and execution history all separate (see also AIP-64: Keep TaskInstance try history). But there is real demand to have support for "experimental approaches" calling for loops, e.g. attempt to train a network until the desired state is reached. For such cases a DAG is not the right thing, a workaround would be to have a long running task that calls a second DAG in a loop.

Continuing to have the option of using the many thousands of operators with 90+ providers

Allowing to scale and complexity of DAGs we have with Airflow 2.x today