Skip to content

Add Python <=> Airflow compat filtering for breeze #52386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 28, 2025
Merged

Add Python <=> Airflow compat filtering for breeze #52386

merged 1 commit into from
Jun 28, 2025

Conversation

amoghrajesh
Copy link
Contributor

One of the teething effects of dropping python 3.9 support for airflow.

The release management commands have a command to generate constraints which run into this error:

"""
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/runner/work/airflow/airflow/dev/breeze/src/airflow_breeze/utils/provider_dependencies.py", line 80, in generate_providers_metadata_for_package
    if constraints[airflow_version].get(package_name) == provider_version:
KeyError: '2.0.0'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/.local/bin/breeze", line 10, in <module>
    sys.exit(main())
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/rich_click/rich_command.py", line 404, in __call__
    return super().__call__(*args, **kwargs)
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/rich_click/rich_command.py", line 187, in main
    rv = self.invoke(ctx)
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/runner/.local/share/uv/tools/apache-airflow-breeze/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/runner/work/airflow/airflow/dev/breeze/src/airflow_breeze/commands/release_management_commands.py", line 2832, in generate_providers_metadata
    results = pool.map(
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
KeyError: '2.0.0'

The hidden reason for this is, python 3.10 support was for Airflow >= 2.4.0 and when we try to download constraints for all the Airflow versions (2.0.x through 2.3.x fail leading to KeyError in the map)

Downloading constraints for all Airflow versions for Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.0 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.0.0-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.0 has not been found. Skipping
  Could not download constraints for Airflow 2.0.0 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.1 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.0.1-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.1 has not been found. Skipping
  Could not download constraints for Airflow 2.0.1 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.2 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.0.2-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.0.2 has not been found. Skipping
  Could not download constraints for Airflow 2.0.2 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.0 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.1.0-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.0 has not been found. Skipping
  Could not download constraints for Airflow 2.1.0 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.1 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.1.1-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.1 has not been found. Skipping
  Could not download constraints for Airflow 2.1.1 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.2 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.1.2-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.2 has not been found. Skipping
  Could not download constraints for Airflow 2.1.2 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.3 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.1.3-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.3 has not been found. Skipping
  Could not download constraints for Airflow 2.1.3 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.4 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.1.4-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.1.4 has not been found. Skipping
  Could not download constraints for Airflow 2.1.4 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.0 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.0-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.0 has not been found. Skipping
  Could not download constraints for Airflow 2.2.0 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.1 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.1-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.1 has not been found. Skipping
  Could not download constraints for Airflow 2.2.1 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.2 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.2-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.2 has not been found. Skipping
  Could not download constraints for Airflow 2.2.2 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.3 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.3-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.3 has not been found. Skipping
  Could not download constraints for Airflow 2.2.3 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.4 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.4-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.4 has not been found. Skipping
  Could not download constraints for Airflow 2.2.4 and Python 3.10
  Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.5 to /home/runner/work/airflow/airflow/.build/constraints/constraints-2.2.5-python-3.10.txt
  The https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.2.5 has not been found. Skipping
  Could not download constraints for Airflow 2.2.5 and Python 3.10

Added a mechanism to detect the compatibility and fixing it by filtering the min airflow version supported for a current python version. Will be useful in long run too. (cc @potiuk maybe for 3.13 support)

After changes:

Fetching all released Airflow 2/3 versions from GitHub

All Airflow 2 versions
  2.0.0: 2020-12-17T17:18:52Z
  2.0.1: 2021-02-08T22:46:44Z
  2.0.2: 2021-04-19T21:05:42Z
  2.1.0: 2021-05-18T10:51:10Z
  2.1.1: 2021-07-02T19:35:41Z
  2.1.2: 2021-07-10T08:47:14Z
  2.1.3: 2021-08-23T16:49:54Z
  2.1.4: 2021-09-18T21:57:29Z
  2.2.0: 2021-10-11T17:18:38Z
  2.2.1: 2021-10-29T17:11:05Z
  2.2.2: 2021-11-15T16:49:03Z
  2.2.3: 2021-12-21T16:25:02Z
  2.2.4: 2022-02-22T20:05:11Z
  2.2.5: 2022-04-04T07:28:42Z
  2.3.0: 2022-04-30T22:10:34Z
  2.3.1: 2022-05-25T07:46:22Z
  2.3.2: 2022-06-04T13:21:00Z
  2.3.3: 2022-07-09T15:51:21Z
  2.3.4: 2022-08-23T11:54:58Z
  2.4.0: 2022-09-19T07:56:24Z
  2.4.1: 2022-09-30T20:05:57Z
  2.4.2: 2022-10-24T09:27:31Z
  2.4.3: 2022-11-14T13:35:43Z
  2.5.0: 2022-12-02T16:35:12Z
  2.5.1: 2023-01-20T18:12:01Z
  2.5.2: 2023-03-14T23:03:15Z
  2.5.3: 2023-03-31T22:22:02Z
  2.6.0: 2023-04-30T12:10:46Z
  2.6.1: 2023-05-16T13:18:41Z
  2.6.2: 2023-06-17T08:13:09Z
  2.6.3: 2023-07-10T20:51:29Z
  2.7.0: 2023-08-18T10:27:22Z
  2.7.1: 2023-09-07T16:49:36Z
  2.7.2: 2023-10-12T09:11:10Z
  2.7.3: 2023-11-06T05:46:51Z
  2.8.0: 2023-12-18T17:06:07Z
  2.8.1: 2024-01-19T10:52:13Z
  2.8.2: 2024-02-26T06:08:13Z
  2.8.3: 2024-03-11T11:09:23Z
  2.8.4: 2024-03-25T20:13:45Z
  2.9.0: 2024-04-08T09:13:53Z
  2.9.1: 2024-05-06T07:05:53Z
  2.9.2: 2024-06-10T08:26:32Z
  2.9.3: 2024-07-16T07:14:53Z
  2.10.0: 2024-08-15T21:08:17Z
  2.10.1: 2024-09-06T07:50:14Z
  2.10.2: 2024-09-20T19:55:14Z
  2.10.3: 2024-11-05T11:03:28Z
  2.10.4: 2024-12-16T10:17:09Z
  2.10.5: 2025-02-10T10:04:09Z
  2.11.0: 2025-05-15T18:42:01Z
  3.0.0: 2025-04-22T13:47:30Z
  3.0.1: 2025-05-12T10:52:44Z
  3.0.2: 2025-06-10T14:05:01Z
Filtering to only use airflow versions supported by current python version: 3.10

Downloading constraints for all Airflow versions for Python 3.10

Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.0 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.0-python-3.10.txt
Downloaded https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.0 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.0-python-3.10.txt
Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.1 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.1-python-3.10.txt
Downloaded https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.1 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.1-python-3.10.txt
Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.2 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.2-python-3.10.txt
Downloaded https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.2 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.2-python-3.10.txt
Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.3 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.3-python-3.10.txt
Downloaded https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.4.3 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.4.3-python-3.10.txt
Downloading https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.5.0 to 
/Users/amoghdesai/Documents/OSS/repos/airflow/.build/constraints/constraints-2.5.0-python-3.10.txt
Downloaded https://api.github.com/repos/apache/airflow/contents/constraints-3.10.txt?ref=constraints-2.5.0 to

Notice we never do for < 2.4.0.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch labels Jun 28, 2025
@amoghrajesh amoghrajesh requested a review from eladkal June 28, 2025 05:50
@amoghrajesh amoghrajesh self-assigned this Jun 28, 2025
amoghrajesh
amoghrajesh commented Jun 28, 2025
View reviewed changes
@potiuk potiuk merged commit 3ef4744 into apache:main Jun 28, 2025
95 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 28, 2025
@amoghrajesh
[v3-0-test] Add Python <=> Airflow compat filtering for breeze (#52386)
b18b4f8 
(cherry picked from commit 3ef4744)

Co-authored-by: Amogh Desai <[email protected]>
@github-actions GitHub Actions
Copy link

Backport successfully created: v3-0-test

Status Branch Result
✅ v3-0-test PR Link

@gopidesupavan
Copy link
Member

nice thanks..

potiuk pushed a commit that referenced this pull request Jun 28, 2025
@amoghrajesh @potiuk
[v3-0-test] Add Python <=> Airflow compat filtering for breeze (#52386)
f5b2e89 
(cherry picked from commit 3ef4744)

Co-authored-by: Amogh Desai <[email protected]>
potiuk pushed a commit that referenced this pull request Jun 28, 2025
@github-actions @amoghrajesh
[v3-0-test] Add Python <=> Airflow compat filtering for breeze (#52386)…
dd70201 
… (#52390)

(cherry picked from commit 3ef4744)

Co-authored-by: Amogh Desai <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potiuk potiuk potiuk approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

@eladkal eladkal Awaiting requested review from eladkal

Assignees

@amoghrajesh amoghrajesh

Labels
area:dev-tools backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@amoghrajesh @gopidesupavan @potiuk