Skip to content

Add credential configuration file support to Google Cloud Hook #31548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 26, 2023
Merged

Add credential configuration file support to Google Cloud Hook #31548

merged 7 commits into from
May 26, 2023

Conversation

pgagnon
Copy link
Contributor

@pgagnon pgagnon commented May 25, 2023

Add support to authenticate to GCP using a credential configuration file explicitly defined in a connection.

This allows Airflow users to authenticate to GCP using external accounts without relying on the ADC mechanism, allowing the configuration of multiple connections utilizing this mechanism, which offers more flexibility than service account keys.

@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels May 25, 2023
@pgagnon pgagnon force-pushed the gcp_credentials_file branch from 110230f to f896c01 Compare May 25, 2023 16:22
ashb
ashb reviewed May 25, 2023
View reviewed changes
Copy link
Member

@ashb ashb left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too familiar with Google creds, so this might be a stupid q, but how is this student to the key_path option?

@pgagnon
Copy link
Contributor Author

pgagnon commented May 25, 2023

I'm not too familiar with Google creds, so this might be a stupid q, but how is this student to the key_path option?

@ashb This method is a generalization that supports a wider range of file-based credentials (authorized_user, service_account, external_account, external_account_authorized_user, impersonated_service_account, gdch_service_account), while key_path only supports long-lived service account keys.

@ashb
Copy link
Member

ashb commented May 25, 2023

Do we need both? Should we deprecated the key one? Can we "just" pass the other types to key_path?

@pgagnon
Copy link
Contributor Author

pgagnon commented May 25, 2023

@ashb We could probably deprecate it, yes, but:

  1. There are just a bunch of cases that I haven't tested related to how we currently handle delegation/impersonation.
  2. google.auth.load_credentials_from_file doesn't support passing a JSON dict directly, although we could make it work with a temp file.

@ashb
Copy link
Member

ashb commented May 25, 2023

Cool let's leave it for now

@pgagnon pgagnon force-pushed the gcp_credentials_file branch 2 times, most recently from b07a751 to bdb28a9 Compare May 25, 2023 17:55
@pgagnon pgagnon force-pushed the gcp_credentials_file branch from bdb28a9 to 6d3b45f Compare May 25, 2023 21:09
@dstandish dstandish merged commit ef40148 into apache:main May 26, 2023
@pgagnon pgagnon mentioned this pull request May 29, 2023
Merged
@pgagnon pgagnon deleted the gcp_credentials_file branch May 30, 2023 15:06
@eladkal eladkal mentioned this pull request Jun 20, 2023
Closed
86 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phanikumv phanikumv phanikumv left review comments

@ashb ashb ashb approved these changes

@dstandish dstandish dstandish approved these changes

Assignees
No one assigned
Labels
area:providers kind:documentation provider:google Google (including GCP) related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pgagnon @ashb @dstandish @phanikumv