Skip to content

Honor index_urls when venv is created with uv in PythonVirtualenvOperator #52287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

wolfdn
Copy link
Contributor

@wolfdn wolfdn commented Jun 26, 2025

Description

The PythonVirtualenvOperator offers the option to pass additional index_urls to load Python packages from.
It installs those Python packages either using pip or uv. Which one to use is decided automatically depending on the system environment.

The index_urlsare put into a pip.conf file when the Python virtual environment is created. pip is recognizing the configuration from that file automatically - however uv is (currently) not. See: astral-sh/uv#1404

This means that if a user provides index_urls when using the PythonVirtualenvOperator, this may or may not work - depending on whether uv is available on the system.

This PR fixes this issue by adding the providedindex_urls as --extra-index-url arguments to uv (as described here: https://docs.astral.sh/uv/concepts/indexes/#-index-url-and-extra-index-url)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Probably during time of UV integration we missed testing with a external index... thanks for reporting and providing the patch!

I just have two requests - which unfortunately would add a bit of complexity but...
(1) When we initially integrated the support for other index URLs we explicitly did not add the index/extra index URLs to the command line but decided to write the pip-conf to ensure that the command line will not spoil/expose secrets in a shared environment because credentials will be in command line. Even though secrets are usually masked in logs still there is a residual risk that other traces in logs would spoil credentials.
As it is a bit sad that UV does not pick-up the pip.cinf, the other alternative described is using ENV variables to define indexes and credentials (above the linked chapter in https://docs.astral.sh/uv/concepts/indexes/#defining-an-index e.g. via UV_INDEX). Can you change the PR to use ENV's and prevent adding CLI parameters?

(2) I see with the change the passed index URLs are added as extra-index URLs - but for PIP we made it explicitly that they are not "just extra" but the first one listed is the primary and then further are going to be extra, otherwise maybe private package lookups would be exposed to pypi.org. So I would kindly request making this for UV in the same way, first index_url should be the primary site and the next ones are extra.

Ah and I assume the CI failure is un-related to this PR...

@jscheffl
Copy link
Contributor

Ah, and forgot - can you please also add a pytest?

@wolfdn wolfdn force-pushed the bugfix/fix-index-urls-when-venv-is-created-with-uv branch from bbc57f0 to fb6702c Compare June 30, 2025 08:42
@wolfdn
Copy link
Contributor Author

wolfdn commented Jun 30, 2025

Thanks for the PR! Probably during time of UV integration we missed testing with a external index... thanks for reporting and providing the patch!

I just have two requests - which unfortunately would add a bit of complexity but... (1) When we initially integrated the support for other index URLs we explicitly did not add the index/extra index URLs to the command line but decided to write the pip-conf to ensure that the command line will not spoil/expose secrets in a shared environment because credentials will be in command line. Even though secrets are usually masked in logs still there is a residual risk that other traces in logs would spoil credentials. As it is a bit sad that UV does not pick-up the pip.cinf, the other alternative described is using ENV variables to define indexes and credentials (above the linked chapter in https://docs.astral.sh/uv/concepts/indexes/#defining-an-index e.g. via UV_INDEX). Can you change the PR to use ENV's and prevent adding CLI parameters?

(2) I see with the change the passed index URLs are added as extra-index URLs - but for PIP we made it explicitly that they are not "just extra" but the first one listed is the primary and then further are going to be extra, otherwise maybe private package lookups would be exposed to pypi.org. So I would kindly request making this for UV in the same way, first index_url should be the primary site and the next ones are extra.

Ah and I assume the CI failure is un-related to this PR...

Thanks for the review! 😊
I implemented the following points:

  • Passing the index URLs to uv using environment variables now instead of command line arguments
  • Changed the implementation so that the first index URL is used as UV_DEFAULT_INDEX and additional index URLs are used as UV_INDEX (this is the same as UV_EXTRA_INDEX_URL, which is deprecated).
  • Updated documentation to explain handling of multiple index URLs
  • Adapted unit tests

@wolfdn wolfdn marked this pull request as ready for review June 30, 2025 08:49
@potiuk potiuk merged commit 0cc667e into apache:main Jul 1, 2025
101 checks passed
@potiuk
Copy link
Member

potiuk commented Jul 1, 2025

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants