THIS PAGE IS ARCHIVED AND IS THERE FOR HISTOROCAL REFERENCE ONLY.
The docs for backport packages are available in https://github.com/apache/airflow/blob/master/README.md#backport-packages and System Tests work will be carried on as part of AIP-4 Automation of System Tests [Deps: AIP-47]
We are providing backported providers packages for Airlow 1.10.* series.
Airflow 2.0 introduced a number of incompatibilities for Airlfow 1.10.* series and we want to make it possible to incrementally migrate to Airflow 2.0 when it gets released. That's why we are going to release Airflow 2.0 operators/sensors and hooks for external systems to make it possible for the users to start using the Airflow 2.0 hooks/operators/sensors in 1.10.* environment. Since in Airflow 2.0 all the non-core operators/hooks/sensors were moved to "airflow.providers" package (following AIP-21: Changes in import paths proposal) they are usable in Airflow 1.10.* environment providing that python 3.6+ is used as python interpreter.
We propose the following migration path for the users of older Airflow versions:
1) If you are using python 2.* or 3.5 or below - migrate your python version to 3.6+ first. This should be rather painless migration.
2) If you are already using python 3.6 or above - install the latest apache-airflow-PACKAGE_NAME pip packages (not yet officially released) and switch incrementally to 'airflow.providers' operators and hooks in your DAGS. We will provide tools to test if your DAGs are using only the new operators so that you can track the progress of your migration
3) Once you switched to new "providers" operators completely and apache-airflow 2.0 is released you should be able to switch to the new airflow version without having to modify your DAGS.
Status of System Tested packages (all package names are : airflow-providers-PACKAGE_NAME
| Backport package name | Release status | Versions tested | Maintainers | Comments |
|---|---|---|---|---|
| Testing completed 100% | 1.10.9 1.10.6 | Details in the document | ||
| postgres | Testing completed 100% | |||
| mysql | Testing completed 100% | |||
| microsoft | Looking for volunteers | |||
| amazon | Testing + Looking for more volunteers | |||
| salesforce | Looking for volunteers | |||
| snowflake | Testing | |||
| http | Looking for volunteers | |||
| apache-spark | Looking for volunteers | |||
| cncf-kubernetes | Testing | Roi Teveth | ||
| presto | Testing completed - 100% operators |
8 Comments
Kamil Bregula
What is the status of extension for AIP ?
If you are planning to do any other work, could I ask you to create **one main issue**, which will allow us to track the progress of work. When we're done, we can close the ticket. All other AIPs have a ticket like this, and I think we should stick to it.
If we've done all the work, can I request a status update for this extension for AIP. I would be happy if at the beginning of this document there was a table with current status, discussion thread, Link to Jira/Github issue.
Jarek Potiuk
I think this page served it purpose are temporary state "capturing" but most of it has been already moved to the right documentation. Also AIP-4 is where most of the work described here has been moved to and it should be treated when we get to the stage of implementing it. It has lower priority than most of the AIP's that are needed to make 2.0 release, so we should recreate it/come back to it when we accept/start working on AIP-4. I am archiving this page now.
Kamil Bregula
So you are not planning to work on new packages in any way and you would not like someone to help you with that? I think it's worth having a ticket that describes what we do, how we do and how new people can contribute to it. I think this is the main goal that every AIP should have a ticket.
Jarek Potiuk
No. i am planning to work on Backport packages and there is already an issue for that : https://github.com/apache/airflow/issues/10014
Simply this page is outdated. And yes. once the AIP-4 gets approved and we vote on it, it will get its own set of tickets - no sooner, no later. There are a lot of questions to be answered still before we define scope for it in sufficient level of details.
Kamil Bregula
Could you please add the current status at the top of this page? Unfortunately, archival pages are not visibly tagged in Jira, so newcomers may overlook that this document is no longer maintained. On the other hand, it is worth pointing to a new place where the user can get current information, e.g. link to the pip repository. Not everyone follows the mailing list. I am trying to break down barriers for new contributors to facilitate the release of Airfllow 2.0
Jarek Potiuk
Kamil Bregula Sure. Done. It would be great if someone reviews all old pages and updates status for all the old and not valid pages. Maybe you go trough them and notify the authors? This way we could cleanup the wiki as it is quite messy now. Just updating one page will not help too much for a new contributor.
Or maybe we can implement what I proposed in https://github.com/apache/airflow/issues/10085#issuecomment-667622228 and move the wiki to Github Issues and make them organized there well and make sure old pages are not there.
Kamil Bregula
I've already reached out to a few contributors who have their AIP on the wiki and asked for updates.
Re: AIP-31: Airflow functional DAG definition
Re: AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance
Re: AIP-3 Drop support for Python 2
Re: AIP-26 Production-ready Airflow Docker Image and helm chart
I focused on the AIPs that are necessary for Airflow 2.0
Jarek Potiuk
Yeah - but I think it does not help with the other pages - if we want to make life of contributors easy, we should cleanup the whole mess because new contributors have no idea which of those pages are important or not.
I'd say we have the opportunity to move only relevant stuff to Github Wiki and keep it there rather than trying to fix the pages in CWiki (similarly as we did with issues). looking forward to trying it.