Skip to content

Commit

Permalink
Remove Backport Providers (#14886)
Browse files Browse the repository at this point in the history 
We are removing support for Backport Providers now.

The last release was sent yesterday- as planned, on 17 March 2021 - the
last release of the Backport Providers.

As agreed before, and documented here:
https://github.com/apache/airflow/blob/master/dev/PROJECT_GUIDELINES.md#support-for-backport-providers

> Backport providers within 1.10.x, will be supported for critical fixes
for three months (March 17, 2021) from Airflow 2.0.0 release date (Dec
17, 2020).

For the future reference, if anyone would like to build backport
providers with cherry-picking any fixes, the branch to start from is
`legacy-backport-cutoff-point`. The documentation and tools to build the
backports are there, but there will be no more community releases for
backports.

Good Bye Backport Providers.
  • Loading branch information
@potiuk
potiuk committed Mar 20, 2021
1 parent 2f32df7 commit 68e4c4d
Show file tree
Hide file tree
Showing 459 changed files with 332 additions and 23,403 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build-images-workflow-run.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ jobs:
sourceRunId: ${{ github.event.workflow_run.id }}
notifyPRCancel: true
jobNameRegexps: >
["^Pylint$", "^Static checks", "^Build docs$", "^Spell check docs$", "^Backport packages$",
["^Pylint$", "^Static checks", "^Build docs$", "^Spell check docs$",
"^Provider packages", "^Checks: Helm tests$", "^Test OpenAPI*"]
- name: "Extract canceled failed runs"
# We use this step to build regexp that will be used to match the Source Run id in
Expand Down
58 changes: 0 additions & 58 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -486,62 +486,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
github.event_name == 'push'
run: aws s3 sync --delete ./files/documentation s3://apache-airflow-docs

prepare-backport-provider-packages:
timeout-minutes: 40
name: "Backport packages: ${{ matrix.package-format }}"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info, ci-images]
strategy:
matrix:
package-format: ['wheel', 'sdist']
env:
# In this case we want to install airflow from the latest released 1.10 version
# all provider packages are installed from wheels or .tar.gz files
INSTALL_AIRFLOW_VERSION: "1.10.14"
AIRFLOW_EXTRAS: "all"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
BACKPORT_PACKAGES: "true"
VERSION_SUFFIX_FOR_PYPI: "dev"
VERSION_SUFFIX_FOR_SVN: "dev"
PACKAGE_FORMAT: ${{ matrix.package-format }}
GITHUB_REGISTRY: ${{ needs.ci-images.outputs.githubRegistry }}
if: needs.build-info.outputs.image-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
with:
persist-credentials: false
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Prepare provider documentation"
run: ./scripts/ci/provider_packages/ci_prepare_provider_documentation.sh
- name: "Prepare provider packages: ${{ matrix.package-format }}"
run: ./scripts/ci/provider_packages/ci_prepare_provider_packages.sh
- name: "Install and test provider packages and airflow via ${{ matrix.package-format }} files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
INSTALL_PROVIDERS_FROM_SOURCES: "false"
- name: "Upload package artifacts"
uses: actions/upload-artifact@v2
if: always()
with:
name: airflow-backport-packages
path: "./dist/apache*"
retention-days: 7
- name: "Upload documentation artifacts"
uses: actions/upload-artifact@v2
if: always() && matrix.package-format == 'wheel'
with:
name: airflow-backport-readmes
path: "./files/airflow-readme-*"
retention-days: 7

prepare-provider-packages:
timeout-minutes: 40
name: "Provider packages ${{ matrix.package-format }}"
Expand Down Expand Up @@ -1112,7 +1056,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- tests-postgres
- tests-mysql
- tests-kubernetes
- prepare-backport-provider-packages
- prepare-provider-packages
- test-provider-packages-released-airflow
- prod-images
Expand Down Expand Up @@ -1162,7 +1105,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- tests-postgres
- tests-mysql
- tests-kubernetes
- prepare-backport-provider-packages
- ci-images
- verify-ci-images
- docs
Expand Down
1 change: 0 additions & 1 deletion .rat-excludes
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ rat-results.txt
apache-airflow-.*\+source.tar.gz.*
apache-airflow-.*\+bin.tar.gz.*
PULL_REQUEST_TEMPLATE.md
BACKPORT_PROVIDER_CHANGES*.md
PROVIDER_CHANGES*.md
manifests/*
redirects.txt
Expand Down
52 changes: 9 additions & 43 deletions BREEZE.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -562,9 +562,6 @@ There are several commands that you can run in Breeze to manage and build packag
Preparing provider readme files is part of the release procedure by the release managers
and it is described in detail in `dev <dev/README.md>`_ .

You can prepare provider packages - by default regular provider packages are prepared, but with
``--backport`` flag you can prepare backport packages.

The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
before running, so you should run it before generating airflow package below as it will be removed.

Expand All @@ -582,13 +579,6 @@ providers that you would like to build. By default ``both`` types of packages ar
./breeze prepare-provider-packages google amazon
You can also prepare backport provider packages, if you specify ``--backport`` flag. You can read more
about backport packages in `dev <dev/README.md>`_

.. code-block:: bash
./breeze prepare-provider-packages --backports google amazon
You can see all providers available by running this command:

.. code-block:: bash
Expand Down Expand Up @@ -1789,10 +1779,6 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: both
--backports
Prepares backport providers rather than regular ones.
-v, --verbose
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
debugging - when you run breeze with --verbose flags you will be able to see the commands
Expand Down Expand Up @@ -2149,7 +2135,7 @@ This is the current syntax for `./breeze <./breeze>`_:
Detailed usage for command: prepare-provider-documentation
breeze prepare-provider-documentation [FLAGS] [YYYY.MM.DD] [PACKAGE_ID ...]
breeze prepare-provider-documentation [FLAGS] [PACKAGE_ID ...]
Prepares documentation files for provider packages.
Expand All @@ -2158,23 +2144,14 @@ This is the current syntax for `./breeze <./breeze>`_:
If no packages are specified, readme for all packages are generated.
If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
You can also specify --backport flag to prepare backport providers documentation and in this
case you can also optionally specify CALVER version as first parameter.
Examples:
'breeze prepare-provider-documentation' or
'breeze prepare-provider-documentation --version-suffix-for-pypi rc1' or
'breeze prepare-provider-documentation --backports 2020.05.10' or
'breeze prepare-provider-documentation --backports 2020.05.10 https google amazon'
'breeze prepare-provider-documentation --version-suffix-for-pypi rc1'
General form:
'breeze prepare-provider-documentation YYYY.MM.DD <PACKAGE_ID> ...'
* YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date
cannot be earlier than the already released version (the script will fail if it
will be). It can be set in the future anticipating the future release date.
'breeze prepare-provider-documentation <PACKAGE_ID> ...'
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
'google' but in several cases, it might be one level deeper separated with
Expand All @@ -2183,7 +2160,7 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
-S, --version-suffix-for-pypi SUFFIX
Adds optional suffix to the version in the generated backport package. It can be used
Adds optional suffix to the version in the generated provider package. It can be used
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-N, --version-suffix-for-svn SUFFIX
Expand All @@ -2200,10 +2177,6 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: both
--backports
Prepares backport providers rather than regular ones.
-v, --verbose
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
debugging - when you run breeze with --verbose flags you will be able to see the commands
Expand All @@ -2225,7 +2198,7 @@ This is the current syntax for `./breeze <./breeze>`_:
breeze prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
Prepares backport packages. You can provide (after --) optional list of packages to prepare.
Prepares provider packages. You can provide (after --) optional list of packages to prepare.
If no packages are specified, readme for all packages are generated. You can specify optional
--version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
--version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
Expand All @@ -2235,9 +2208,6 @@ This is the current syntax for `./breeze <./breeze>`_:
prepared there so make sure you run prepare-provider-packages first,
and prepare-airflow-packages second.
You can also specify --backport flag to prepare backport providers or --package-format to
prepare one or both types of supported formats.
Examples:
'breeze prepare-provider-packages' or
Expand All @@ -2250,7 +2220,7 @@ This is the current syntax for `./breeze <./breeze>`_:
General form:
'breeze prepare-provider-packages [--backports] [--package-format PACKAGE_FORMAT] \
'breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] \
[--version-suffix-for-svn|--version-suffix-for-pypi] <PACKAGE_ID> ...'
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
Expand All @@ -2269,12 +2239,8 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: both
--backports
Prepares backport providers rather than regular ones.
-S, --version-suffix-for-pypi SUFFIX
Adds optional suffix to the version in the generated backport package. It can be used
Adds optional suffix to the version in the generated provider package. It can be used
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-N, --version-suffix-for-svn SUFFIX
Expand Down Expand Up @@ -2790,10 +2756,10 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: All
****************************************************************************************************
Flags for generation of the backport packages
Flags for generation of the provider packages
-S, --version-suffix-for-pypi SUFFIX
Adds optional suffix to the version in the generated backport package. It can be used
Adds optional suffix to the version in the generated provider package. It can be used
to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-N, --version-suffix-for-svn SUFFIX
Expand Down
6 changes: 2 additions & 4 deletions CI.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -265,10 +265,10 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| Version suffix variables |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``VERSION_SUFFIX_FOR_PYPI`` | | | | Version suffix used during backport |
| ``VERSION_SUFFIX_FOR_PYPI`` | | | | Version suffix used during provider |
| | | | | package preparation for PyPI builds. |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``VERSION_SUFFIX_FOR_SVN`` | | | | Version suffix used during backport |
| ``VERSION_SUFFIX_FOR_SVN`` | | | | Version suffix used during provider |
| | | | | package preparation for SVN builds. |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| Git variables |
Expand Down Expand Up @@ -706,8 +706,6 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------+----------------------------------------------+-------+-------+------+
| Spell check docs | Spell check for documentation | Yes | Yes | Yes |
+---------------------------+----------------------------------------------+-------+-------+------+
| Backport packages | Prepares Backport Packages for 1.10 Airflow | Yes | Yes | Yes |
+---------------------------+----------------------------------------------+-------+-------+------+
| Trigger tests | Checks if tests should be triggered | Yes | Yes | Yes |
+---------------------------+----------------------------------------------+-------+-------+------+
| Tests [Pg/Msql/Sqlite] | Run all the Pytest tests for Python code | Yes(2)| Yes | Yes |
Expand Down
17 changes: 2 additions & 15 deletions CONTRIBUTING.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -629,8 +629,8 @@ The dependency list is automatically used during PyPI packages generation.

Cross-dependencies between provider packages are converted into extras - if you need functionality from
the other provider package you can install it adding [extra] after the
``apache-airflow-backport-providers-PROVIDER`` for example:
``pip install apache-airflow-backport-providers-google[amazon]`` in case you want to use GCP
``apache-airflow-providers-PROVIDER`` for example:
``pip install apache-airflow-providers-google[amazon]`` in case you want to use GCP
transfer operators from Amazon ECS.

If you add a new dependency between different providers packages, it will be detected automatically during
Expand Down Expand Up @@ -764,19 +764,6 @@ providers.
You can read more about out approach for tests in `TESTING.rst <TESTING.rst>`_ but here
are some highlights.


Backport providers
------------------

You can also build backport provider packages for Airflow 1.10. They aim to provide a bridge when users
of Airflow 1.10 want to migrate to Airflow 2.0. The backport packages are named similarly to the
provider packages, but with "backport" added:

* ``apache-airflow-backport-provider-*``

Those backport providers are automatically refactored to work with Airflow 1.10.* and have a few
limitations described in those packages.

Dependency management
=====================

Expand Down
2 changes: 1 addition & 1 deletion IMAGES.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg AIRFLOW_SOURCES_FROM="empty" \
--build-arg AIRFLOW_SOURCES_TO="/empty" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack" \
--build-arg ADDITIONAL_PYTHON_DEPS="apache-airflow-backport-providers-odbc \
--build-arg ADDITIONAL_PYTHON_DEPS="apache-airflow-providers-odbc \
azure-storage-blob \
sshtunnel \
google-api-python-client \
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,8 @@ pip install apache-airflow[postgres,google]==2.0.1 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt"
```

For information on installing backport providers check [backport-providers.rst](docs/apache-airflow/backport-providers.rst).
For information on installing provider packages check
[providers](http://airflow.apache.org/docs/apache-airflow-providers/index.html).

## Official source code

Expand Down

0 comments on commit 68e4c4d

Please sign in to comment.