Skip to content

Commit c439ab8

Browse files
potiukpotiuk
authored
Standardize airflow build process and switch to Hatchling build backend (#36537)
This PR changes Airflow installation and build backend to use new standard Python ways of building Python applications. We've been trying to do it for quite a while. Airflow tranditionally has been using complex and convoluted build process based on setuptools and (extremely) custom setup.py file. It survived migration to Airflow 2.0 and splitting Airlfow monorepo into Airflow and Providers, adding pre-installed providers and switching providers to use flit (and follow build standards). So far tooling in Python ecosystme had not been able to fuflill our needs and we refrained to develop our own tooling, but finally with appearance of Hatch (managed by Python Packaging Authority) and few recent advancements there we are finally able to swtich to Python standard ways of managing project dependnecy configuration and project build setup (with a few customizations). This PR makes airflow build process follow those standard PEPs: * Airflow has all build configuration stored in pyproject.toml following PEP 518 which allows any fronted (`pip`, `poetry`, `hatch`, `flit`, or whatever other frontend is used to install required build dependendencies to install Airflow locally and to build distribution pacakges (sdist/wheel) * Hatchling backend follows PEP 517 for standard source tree and build backend implementation that allows to execute the build in a frontend-independent way * We store all project metadata in pyprooject.toml - following PEP 621 where all necessary project metadata components were defined. * We plug-in into Hatchling "editable build" hooks following PEP 660. Hatchling internally builds editable wheel that is used as ephemeral step and communication between backend and frontend (and this ephemeral wheel is used to make editable installation of the projeect - suitable for fast iteration of code without reinstalling the package. With Airflow having many provider packages in single source tree where we want to be able to install and develop airflow and providers together, this is not a small feat to implement the case wher editable installation has to behave quite a bit differently when it comes to packaging and dependencies for editable install (when you want to edit sources directly) and installable package (where you want to have separate Airflow package and provider packages). Fortunately the standardisation efforts in the Python Packaging community and tooling implementing it had finally made it possible. Some of the important ways bow this has been achieved: * We continue using provider.yaml in providers as the single source of trutgh for per-provider dependencies. We added a possibility to specify "devel-dependencies" in provider.yaml so that all per-provider dependencies in `generated/provider_dependencies.json` and `pyproject.toml` are generated from those dependencies via update-providers-dependencies pre-commit. * Pyproject.toml is generally managed manually, but the part where provider dependencies and bundle dependencies are used is automatically updated by a pre-commit whenever provider dependencies change. Those generated provider dependencies contain just dependencies of providers - not the provider packages, but in the final "standard" wheel file they are replaced with "apache-airflow-providers-PROVIDER" dependencies - so that the wheel package will only install the provider and use the dependencies of that version of provider it installs. * We are utilising custom hatchiling build hooks (PEP 660 standard) that allow to modify 'standard' wheel package on-the-fly when the wheel is being prepared by adding preinstalled package dependencies (which are not needed in editable build) and by removing all devel extras (that are not needed in the PyPI distributed wheel package). This allows to solve the conundrum of having different "editable" and "standard" behaviour while keeping the same project specification in pyproject.toml. * We added description of how `Hatch` can be employed as build frontend in order to manage local virtualenv and install Airflow in editable way easily - while keeping all properties of the installed application (including working airflow cli and package metadata discovery) as well as how to use PEP-standard ways of bulding wheel and sdist packages. * We have a custom step (following PEP-standards) to inject airflow-specific build steps - compiling www assets and generating git commit hash version to display it in the UI * We also show how all this makes it possible to make it easy to manage local virtualenvs and editable installations for Airflow contributors - without vendor lock-in of the build tools as by following standard PEPs Airflow can be locally and editably installed by anyone using any build front-end tools following the standards - whether you use `pip`, `poetry`, `Hatch`, `flit` or any other frontent build tools, Airflow local installation and package building will work the same way for all of them, where both "editable" and "standard" package prepration is managed by `hatchling` backend in the same way. * Previously our extras contained a "." which is not normalized name for extras - `pip` and other tools replaced it automatically with `_'. This change updates the extra names to contain '-' rather than '.' in the name, following PEP-685. This should be fully backwards compatible, users will still be able to use "." but it will be normalized to "-" in Airflow packages. This is also future proof as it is expected that all package managers and tools will eventually use PEP-685 applied to extras, even if currently some of the tools (pip + setuptools) might generate warnings. * Additionally, this change organizes the documentation around the extras and dependencies, explaining the reasoning behind all the different extras we have. * As a bonus (and this is what we used to test it all) we are documenting how to use Hatch frontend to: * manage multiple Python installations * manage multiple Pythob virtualenv environments * build Airflow packages for release management
1 parent ead7528 commit c439ab8

File tree

146 files changed

+3650
-3045
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

146 files changed

+3650
-3045
lines changed

‎.dockerignore

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@
4848
!.dockerignore
4949
!RELEASE_NOTES.rst
5050
!LICENSE
51-
!MANIFEST.in
5251
!NOTICE
5352
!.github
5453
!empty
@@ -68,8 +67,6 @@
6867
!.bash_completion.d
6968

7069
# Setup/version configuration
71-
!setup.cfg
72-
!setup.py
7370
!pyproject.toml
7471
!manifests
7572
!generated

‎.github/actions/build-prod-images/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ runs:
4141
shell: bash
4242
run: >
4343
breeze release-management prepare-provider-packages
44-
--package-list-file ./airflow/providers/installed_providers.txt
44+
--package-list-file ./dev/prod_image_installed_providers.txt
4545
--package-format wheel --version-suffix-for-pypi dev0
4646
if: ${{ inputs.build-provider-packages == 'true' }}
4747
- name: "Prepare chicken-eggs provider packages"

‎.github/workflows/ci.yml

Lines changed: 16 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -192,7 +192,7 @@ jobs:
192192

193193
# Push early BuildX cache to GitHub Registry in Apache repository, This cache does not wait for all the
194194
# tests to complete - it is run very early in the build process for "main" merges in order to refresh
195-
# cache using the current constraints. This will speed up cache refresh in cases when setup.py
195+
# cache using the current constraints. This will speed up cache refresh in cases when pyproject.toml
196196
# changes or in case of Dockerfile changes. Failure in this step is not a problem (at most it will
197197
# delay cache refresh. It does not attempt to upgrade to newer dependencies.
198198
# We only push CI cache as PROD cache usually does not gain as much from fresh cache because
@@ -486,7 +486,7 @@ jobs:
486486
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
487487
shell: bash
488488
run: >
489-
breeze release-management prepare-provider-packages
489+
breeze release-management prepare-provider-packages --include-not-ready-providers
490490
--package-format wheel --version-suffix-for-pypi dev0
491491
${{ needs.build-info.outputs.chicken-egg-providers }}
492492
if: needs.build-info.outputs.chicken-egg-providers != ''
@@ -678,9 +678,9 @@ jobs:
678678
id: cache-doc-inventories
679679
with:
680680
path: ./docs/_inventory_cache/
681-
key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
681+
key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
682682
restore-keys: |
683-
docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
683+
docs-inventory-${{ hashFiles('pyproject.toml;') }}
684684
docs-inventory-
685685
- name: "Build docs"
686686
run: >
@@ -742,9 +742,9 @@ jobs:
742742
id: cache-doc-inventories
743743
with:
744744
path: ./docs/_inventory_cache/
745-
key: docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
745+
key: docs-inventory-${{ hashFiles('pyproject.toml;') }}
746746
restore-keys: |
747-
docs-inventory-${{ hashFiles('setup.py','setup.cfg','pyproject.toml;') }}
747+
docs-inventory-${{ hashFiles('pyproject.toml;') }}
748748
docs-inventory-
749749
- name: "Spellcheck docs"
750750
run: >
@@ -773,11 +773,13 @@ jobs:
773773
run: rm -fv ./dist/*
774774
- name: "Prepare provider documentation"
775775
run: >
776-
breeze release-management prepare-provider-documentation --non-interactive
776+
breeze release-management prepare-provider-documentation --include-not-ready-providers
777+
--non-interactive
777778
${{ needs.build-info.outputs.affected-providers-list-as-string }}
778779
- name: "Prepare provider packages: wheel"
779780
run: >
780-
breeze release-management prepare-provider-packages --version-suffix-for-pypi dev0
781+
breeze release-management prepare-provider-packages --include-not-ready-providers
782+
--version-suffix-for-pypi dev0
781783
--package-format wheel ${{ needs.build-info.outputs.affected-providers-list-as-string }}
782784
- name: "Prepare airflow package: wheel"
783785
run: breeze release-management prepare-airflow-package --version-suffix-for-pypi dev0
@@ -846,7 +848,7 @@ jobs:
846848
run: rm -fv ./dist/*
847849
- name: "Prepare provider packages: sdist"
848850
run: >
849-
breeze release-management prepare-provider-packages
851+
breeze release-management prepare-provider-packages --include-not-ready-providers
850852
--version-suffix-for-pypi dev0 --package-format sdist
851853
${{ needs.build-info.outputs.affected-providers-list-as-string }}
852854
- name: "Prepare airflow package: sdist"
@@ -913,7 +915,7 @@ jobs:
913915
run: rm -fv ./dist/*
914916
- name: "Prepare provider packages: wheel"
915917
run: >
916-
breeze release-management prepare-provider-packages
918+
breeze release-management prepare-provider-packages --include-not-ready-providers
917919
--package-format wheel ${{ needs.build-info.outputs.affected-providers-list-as-string }}
918920
- name: >
919921
Remove incompatible Airflow
@@ -922,17 +924,9 @@ jobs:
922924
rm -vf ${{ matrix.remove-providers }}
923925
working-directory: ./dist
924926
if: matrix.remove-providers != ''
925-
- name: "Checkout ${{matrix.airflow-version}} of Airflow"
926-
uses: actions/checkout@v4
927-
with:
928-
persist-credentials: false
929-
ref: ${{matrix.airflow-version}}
930-
path: old-airflow
931-
- name: "Prepare airflow package: wheel"
927+
- name: "Download airflow package: wheel"
932928
run: |
933-
pip install pip==23.3.2 wheel==0.36.2 gitpython==3.1.40
934-
python setup.py egg_info --tag-build ".dev0" bdist_wheel -d ../dist
935-
working-directory: ./old-airflow
929+
pip download "apache-airflow==${{matrix.airflow-version}}" -d dist --no-deps
936930
- name: >
937931
Install and verify all provider packages and airflow on
938932
Airflow ${{matrix.airflow-version}}:Python ${{matrix.python-version}}
@@ -2050,8 +2044,7 @@ jobs:
20502044
path: ".build/.k8s-env"
20512045
key: "\
20522046
k8s-env-${{steps.breeze.outputs.host-python-version}}-\
2053-
${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','setup.cfg',\
2054-
'setup.py','pyproject.toml','generated/provider_dependencies.json') }}"
2047+
${{ hashFiles('scripts/ci/kubernetes/k8s_requirements.txt','pyproject.toml') }}"
20552048
- name: Run complete K8S tests ${{needs.build-info.outputs.kubernetes-combos-list-as-string}}
20562049
run: breeze k8s run-complete-tests --run-in-parallel --upgrade
20572050
env:
@@ -2182,7 +2175,7 @@ jobs:
21822175
- name: "Prepare providers packages for PROD build"
21832176
run: >
21842177
breeze release-management prepare-provider-packages
2185-
--package-list-file ./airflow/providers/installed_providers.txt
2178+
--package-list-file ./dev/prod_image_installed_providers.txt
21862179
--package-format wheel
21872180
env:
21882181
VERSION_SUFFIX_FOR_PYPI: "dev0"

‎.gitignore

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -190,8 +190,6 @@ dmypy.json
190190
log.txt*
191191

192192
# Provider-related ignores
193-
/provider_packages/CHANGELOG.txt
194-
/provider_packages/MANIFEST.in
195193
/airflow/providers/__init__.py
196194

197195
# Docker context files
@@ -219,7 +217,7 @@ pip-wheel-metadata
219217
/dev/Dockerfile.pmc
220218

221219
# Generated UI licenses
222-
licenses/LICENSES-ui.txt
220+
3rd-party-licenses/LICENSES-ui.txt
223221

224222
# Packaged breeze on Windows
225223
/breeze.exe
@@ -240,3 +238,6 @@ licenses/LICENSES-ui.txt
240238

241239
# Dask Executor tests generate this directory
242240
/tests/executors/dask-worker-space/
241+
242+
# airflow-build-dockerfile and correconding ignore file
243+
airflow-build-dockerfile*

‎.pre-commit-config.yaml

Lines changed: 20 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -274,11 +274,6 @@ repos:
274274
- --ignore-words=docs/spelling_wordlist.txt
275275
- --skip=airflow/providers/*/*.rst,airflow/www/*.log,docs/*/commits.rst,docs/apache-airflow/tutorial/pipeline_example.csv,*.min.js,*.lock,INTHEWILD.md
276276
- --exclude-file=.codespellignorelines
277-
- repo: https://github.com/abravalheri/validate-pyproject
278-
rev: v0.15
279-
hooks:
280-
- id: validate-pyproject
281-
name: Validate pyproject.toml
282277
- repo: local
283278
# Note that this is the 2nd "local" repo group in the .pre-commit-config.yaml file. This is because
284279
# we try to minimise the number of passes that must happen in order to apply some of the changes
@@ -333,13 +328,6 @@ repos:
333328
files: Dockerfile.*$
334329
pass_filenames: true
335330
require_serial: true
336-
- id: check-setup-order
337-
name: Check order of dependencies in setup.cfg and setup.py
338-
language: python
339-
files: ^setup\.cfg$|^setup\.py$
340-
pass_filenames: false
341-
entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
342-
additional_dependencies: ['rich>=12.4.4']
343331
- id: check-airflow-k8s-not-used
344332
name: Check airflow.kubernetes imports are not used
345333
language: python
@@ -363,14 +351,6 @@ repos:
363351
exclude: ^airflow/kubernetes/|^airflow/providers/
364352
entry: ./scripts/ci/pre_commit/pre_commit_check_cncf_k8s_used_for_k8s_executor_only.py
365353
additional_dependencies: ['rich>=12.4.4']
366-
- id: check-extra-packages-references
367-
name: Checks setup extra packages
368-
description: Checks if all the libraries in setup.py are listed in extra-packages-ref.rst file
369-
language: python
370-
files: ^setup\.py$|^docs/apache-airflow/extra-packages-ref\.rst$|^airflow/providers/.*/provider\.yaml$
371-
pass_filenames: false
372-
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
373-
additional_dependencies: ['rich>=12.4.4']
374354
- id: check-airflow-provider-compatibility
375355
name: Check compatibility of Providers with Airflow
376356
entry: ./scripts/ci/pre_commit/pre_commit_check_provider_airflow_compatibility.py
@@ -400,19 +380,34 @@ repos:
400380
files: ^airflow/providers/.*/hooks/.*\.py$
401381
additional_dependencies: ['rich>=12.4.4', 'pyyaml', 'packaging']
402382
- id: update-providers-dependencies
403-
name: Update cross-dependencies for providers packages
383+
name: Update dependencies for provider packages
404384
entry: ./scripts/ci/pre_commit/pre_commit_update_providers_dependencies.py
405385
language: python
406-
files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$
386+
files: ^airflow/providers/.*\.py$|^airflow/providers/.*/provider\.yaml$|^tests/providers/.*\.py$|^tests/system/providers/.*\.py$|^scripts/ci/pre_commit/pre_commit_update_providers_dependencies\.py$
407387
pass_filenames: false
408-
additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml']
388+
additional_dependencies: ['setuptools', 'rich>=12.4.4', 'pyyaml', 'tomli']
389+
- id: check-extra-packages-references
390+
name: Checks setup extra packages
391+
description: Checks if all the extras defined in pyproject.toml are listed in extra-packages-ref.rst file
392+
language: python
393+
files: ^docs/apache-airflow/extra-packages-ref\.rst$|^pyproject.toml
394+
pass_filenames: false
395+
entry: ./scripts/ci/pre_commit/pre_commit_check_extra_packages_ref.py
396+
additional_dependencies: ['rich>=12.4.4', 'tomli', 'tabulate']
397+
- id: check-pyproject-toml-order
398+
name: Check order of dependencies in pyproject.toml
399+
language: python
400+
files: ^pyproject\.toml$
401+
pass_filenames: false
402+
entry: ./scripts/ci/pre_commit/pre_commit_check_order_pyproject_toml.py
403+
additional_dependencies: ['rich>=12.4.4']
409404
- id: update-extras
410405
name: Update extras in documentation
411406
entry: ./scripts/ci/pre_commit/pre_commit_insert_extras.py
412407
language: python
413408
files: ^setup\.py$|^CONTRIBUTING\.rst$|^INSTALL$|^airflow/providers/.*/provider\.yaml$
414409
pass_filenames: false
415-
additional_dependencies: ['rich>=12.4.4']
410+
additional_dependencies: ['rich>=12.4.4', 'tomli']
416411
- id: check-extras-order
417412
name: Check order of extras in Dockerfile
418413
entry: ./scripts/ci/pre_commit/pre_commit_check_order_dockerfile_extras.py
@@ -712,7 +707,7 @@ repos:
712707
name: Sort alphabetically and uniquify installed_providers.txt
713708
entry: ./scripts/ci/pre_commit/pre_commit_sort_installed_providers.py
714709
language: python
715-
files: ^\.pre-commit-config\.yaml$|^airflow/providers/installed_providers\.txt$
710+
files: ^\.pre-commit-config\.yaml$|^dev/.*_installed_providers\.txt$
716711
pass_filenames: false
717712
require_serial: true
718713
- id: update-spelling-wordlist-to-be-sorted

‎.rat-excludes

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,8 @@ venv
4343
files
4444
airflow.iml
4545
.gitmodules
46-
installed_providers.txt
46+
prod_image_installed_providers.txt
47+
airflow_pre_installed_providers.txt
4748

4849
# Generated doc files
4950
.*html
@@ -61,7 +62,7 @@ spelling_wordlist.txt
6162
# it is compatible according to http://www.apache.org/legal/resolved.html#category-a
6263
kerberos_auth.py
6364
airflow_api_auth_backend_kerberos_auth_py.html
64-
licenses/*
65+
3rd-party-licenses/*
6566
parallel.js
6667
underscore.js
6768
jquery.dataTables.min.js

‎licenses/LICENSE-bootstrap.txt renamed to ‎3rd-party-licenses/LICENSE-bootstrap.txt

File renamed without changes.

‎licenses/LICENSE-bootstrap3-typeahead.txt renamed to ‎3rd-party-licenses/LICENSE-bootstrap3-typeahead.txt

File renamed without changes.

‎licenses/LICENSE-d3-shape.txt renamed to ‎3rd-party-licenses/LICENSE-d3-shape.txt

File renamed without changes.

‎licenses/LICENSE-d3-tip.txt renamed to ‎3rd-party-licenses/LICENSE-d3-tip.txt

File renamed without changes.

‎licenses/LICENSE-d3js.txt renamed to ‎3rd-party-licenses/LICENSE-d3js.txt

File renamed without changes.

‎licenses/LICENSE-dagre-d3.txt renamed to ‎3rd-party-licenses/LICENSE-dagre-d3.txt

File renamed without changes.

‎licenses/LICENSE-datatables.txt renamed to ‎3rd-party-licenses/LICENSE-datatables.txt

File renamed without changes.

‎licenses/LICENSE-elasticmock.txt renamed to ‎3rd-party-licenses/LICENSE-elasticmock.txt

File renamed without changes.

‎licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt renamed to ‎3rd-party-licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt

File renamed without changes.

‎licenses/LICENSE-flask-kerberos.txt renamed to ‎3rd-party-licenses/LICENSE-flask-kerberos.txt

File renamed without changes.

‎licenses/LICENSE-hue.txt renamed to ‎3rd-party-licenses/LICENSE-hue.txt

File renamed without changes.

‎licenses/LICENSE-jqclock.txt renamed to ‎3rd-party-licenses/LICENSE-jqclock.txt

File renamed without changes.

‎licenses/LICENSE-jquery.txt renamed to ‎3rd-party-licenses/LICENSE-jquery.txt

File renamed without changes.

‎licenses/LICENSE-moment.txt renamed to ‎3rd-party-licenses/LICENSE-moment.txt

File renamed without changes.

‎licenses/LICENSE-normalize.txt renamed to ‎3rd-party-licenses/LICENSE-normalize.txt

File renamed without changes.

‎licenses/LICENSE-pytest-capture-warnings.txt renamed to ‎3rd-party-licenses/LICENSE-pytest-capture-warnings.txt

File renamed without changes.

‎licenses/LICENSE-unicodecsv.txt renamed to ‎3rd-party-licenses/LICENSE-unicodecsv.txt

File renamed without changes.

‎BREEZE.rst

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1569,7 +1569,7 @@ The CI image is built automatically as needed, however it can be rebuilt manuall
15691569
15701570
Building the image first time pulls a pre-built version of images from the Docker Hub, which may take some
15711571
time. But for subsequent source code changes, no wait time is expected.
1572-
However, changes to sensitive files like ``setup.py`` or ``Dockerfile.ci`` will trigger a rebuild
1572+
However, changes to sensitive files like ``pyproject.toml`` or ``Dockerfile.ci`` will trigger a rebuild
15731573
that may take more time though it is highly optimized to only rebuild what is needed.
15741574
15751575
Breeze has built in mechanism to check if your local image has not diverged too much from the
@@ -2299,7 +2299,7 @@ These are all available flags of ``release-management add-back-references`` comm
22992299
Generating constraints
23002300
""""""""""""""""""""""
23012301
2302-
Whenever setup.py gets modified, the CI main job will re-generate constraint files. Those constraint
2302+
Whenever ``pyproject.toml`` gets modified, the CI main job will re-generate constraint files. Those constraint
23032303
files are stored in separated orphan branches: ``constraints-main``, ``constraints-2-0``.
23042304
23052305
Those are constraint files as described in detail in the
@@ -2341,14 +2341,14 @@ These are all available flags of ``generate-constraints`` command:
23412341
:width: 100%
23422342
:alt: Breeze generate-constraints
23432343
2344-
In case someone modifies setup.py, the scheduled CI Tests automatically upgrades and
2344+
In case someone modifies ``pyproject.toml``, the scheduled CI Tests automatically upgrades and
23452345
pushes changes to the constraint files, however you can also perform test run of this locally using
23462346
the procedure described in the
23472347
`Manually generating image cache and constraints <dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md>`_
23482348
which utilises multiple processors on your local machine to generate such constraints faster.
23492349
2350-
This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
2351-
and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
2350+
This bumps the constraint files to latest versions and stores hash of ``pyproject.toml``. The generated constraint
2351+
and ``pyproject.toml`` hash files are stored in the ``files`` folder and while generating the constraints diff
23522352
of changes vs the previous constraint files is printed.
23532353
23542354
Updating constraints
@@ -2697,18 +2697,18 @@ disappear when you exit Breeze shell.
26972697
26982698
When you want to add dependencies permanently, then it depends what kind of dependency you add.
26992699
2700-
If you want to add core dependency that should always be installed - you need to add it to ``setup.cfg``
2701-
to ``install_requires`` section. If you want to add it to one of the optional core extras, you should
2702-
add it in the extra definition in ``setup.py`` (you need to find out where it is defined). If you want
2703-
to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
2700+
If you want to add core dependency that should always be installed - you need to add it to ``pyproject.toml``
2701+
to ``dependencies`` section. If you want to add it to one of the optional core extras, you should
2702+
add it in the extra definition in ``pyproject.toml`` (you need to find out where it is defined).
2703+
If you want to add it to one of the providers, you need to add it to the ``provider.yaml`` file in the provider
27042704
directory - but remember that this should be followed by running pre-commit that will automatically update
2705-
the ``generated/provider_dependencies.json`` directory with the new dependencies:
2705+
the ``pyproject.toml`` with the new dependencies as the ``provider.yaml`` files are not used directly, they
2706+
are used to update ``pyproject.toml`` file:
27062707
27072708
.. code-block:: bash
27082709
27092710
pre-commit run update-providers-dependencies --all-files
27102711
2711-
27122712
You can also run the pre-commit by ``breeze static-checks --type update-providers-dependencies --all-files``
27132713
command - which provides autocomplete.
27142714

‎CI.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -617,7 +617,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
617617
| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | false | false | false\* | Determines whether the build should |
618618
| | | | | attempt to upgrade Python base image and all |
619619
| | | | | PIP dependencies to latest ones matching |
620-
| | | | | ``setup.py`` limits. This tries to replicate |
620+
| | | | | ``pyproject.toml`` limits. Tries to replicate |
621621
| | | | | the situation of "fresh" user who just installs |
622622
| | | | | airflow and uses latest version of matching |
623623
| | | | | dependencies. By default we are using a |
@@ -638,7 +638,7 @@ those via corresponding command line flags passed to ``breeze shell`` command.
638638
| | | | | |
639639
| | | | | Setting the value to random value is best way |
640640
| | | | | to assure that constraints are upgraded even if |
641-
| | | | | there is no change to setup.py |
641+
| | | | | there is no change to ``pyproject.toml`` |
642642
| | | | | |
643643
| | | | | This way our constraints are automatically |
644644
| | | | | tested and updated whenever new versions |

0 commit comments

Comments
 (0)