Lazily import fs and package_index hook in providers manager #52117
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I discovered this side-effect that once you call `ProvidersManager()._clear()` then the `fs` connection type was no longer registered. This is fixed by moving the call to `_init_airflow_core_hooks` into the (already cached in memory once per instance) `initialize_providers_hooks` function This lead me to notice that we were directly importing the standard provider eagerly at the top level, which we want to avoid in Airflow 3 with the work torward separation of Core and Task SDK. So I changed it to make use of the existing "lazy resolve" feature of `_hooks_lazy_dict`. And finally, I removed the `_hooks_dict` attribute which was being created, but never looked at again.
jedcunningham
approved these changes
Jun 23, 2025
|
Nice! Thanks @ashb ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I discovered this side-effect that once you call
ProvidersManager()._clear()then the
fsconnection type was no longer registered. This is fixed bymoving the call to
_init_airflow_core_hooksinto the (already cached inmemory once per instance)
initialize_providers_hooksfunctionThis lead me to notice that we were directly importing the standard provider
eagerly at the top level, which we want to avoid in Airflow 3 with the work
torward separation of Core and Task SDK. So I changed it to make use of the
existing "lazy resolve" feature of
_hooks_lazy_dict.And finally, I removed the
_hooks_dictattribute which was being created,but never looked at again.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.