Skip to content

Snowflake SQL API hook Retry Logic #51463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Jun 18, 2025
Merged

Snowflake SQL API hook Retry Logic #51463

merged 10 commits into from
Jun 18, 2025
+518 −60

Conversation

astro-anand
Copy link
Contributor

@astro-anand astro-anand commented Jun 5, 2025 •
edited
Loading

Updated sync and async Snowflake API call methods to retry on errors as advised by Snowflake (https://docs.snowflake.com/en/developer-guide/sql-api/handling-errors) and on connection related errors.

closes: #50514

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:snowflake Issues related to Snowflake provider labels Jun 5, 2025
@astro-anand astro-anand mentioned this pull request Jun 9, 2025
Closed
2 tasks
@astro-anand
Copy link
Contributor Author

@mik-laj or @potiuk, would it be possible to get a review of this soon? I'm working with a few teams that heavily use the Snowflake SQL API and this would help their DAGs run much more smoothly!

@rawwar rawwar self-requested a review June 10, 2025 05:30
mik-laj
mik-laj reviewed Jun 10, 2025
View reviewed changes
@astro-anand astro-anand requested a review from mik-laj June 12, 2025 15:04
@astro-anand
Copy link
Contributor Author

Hey @mik-laj, would you be able to take another look at this soon? Thank you!

@astro-anand
Copy link
Contributor Author

Hey @mik-laj, I've updated the implementation to use context managers for both the http session and the retrying object. Thank you for the feedback and let me know what you think of the updates 🙌

@astro-anand astro-anand force-pushed the snowflake_sql_api_hook_retry_logic branch from 7ddc037 to d892811 Compare June 18, 2025 15:44
@astro-anand astro-anand force-pushed the snowflake_sql_api_hook_retry_logic branch 2 times, most recently from 6ea7b09 to e44cd77 Compare June 18, 2025 19:04
@astro-anand
Copy link
Contributor Author

Image 6-18-25 at 2 09 PM
integration test

@astro-anand astro-anand force-pushed the snowflake_sql_api_hook_retry_logic branch from e44cd77 to 088fc17 Compare June 18, 2025 19:14
@RNHTTR RNHTTR merged commit 25be79a into apache:main Jun 18, 2025
66 checks passed
@astro-anand astro-anand deleted the snowflake_sql_api_hook_retry_logic branch June 18, 2025 19:39
RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Jun 21, 2025
@astro-anand @RoyLee1224
Snowflake SQL API hook Retry Logic (apache#51463)
322ddd5 
* added retry logic for sync and async requests to snowflake api

* first draft of unit tests

* unit tests for hook and retries

* remove comment

* update sync request to use request.request

* mypy fixes

* updated sync and async api call methods to use tenacity context manager

* update unit test with correct method

* retry args docs

* reorder so self.log is initialized
@SeiilkanM
Copy link

Hi @astro-anand do you happen to know when this will be released?

@potiuk
Copy link
Member

potiuk commented Jun 23, 2025 •
edited
Loading

Hi @astro-anand do you happen to know when this will be released?

We usually release provider's every 2 weeks (See the README). We have one release in progress (see devlist) so you might expect the next wave within 2 weeks.

But you can also - and we encourage you to - to build your own provider, it's super easy (there are a number of ways - starting from breeze and ending with just running flit build in the providers folder (this is just absolutely standard python package with it's pyproject.toml`) - and installing it locally for tests.

Also when voting on the new provider starts (you should observe devlist and there will be github issue) you are encouraged to test and see the RC candidate (and let us know when you did)

@AryanK1511 AryanK1511 mentioned this pull request Jun 23, 2025
Merged
@eladkal eladkal mentioned this pull request Jun 24, 2025
Closed
2 tasks
@AryanK1511 AryanK1511 mentioned this pull request Jun 25, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mik-laj mik-laj mik-laj approved these changes

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@rawwar rawwar Awaiting requested review from rawwar

Assignees
No one assigned
Labels
area:providers provider:snowflake Issues related to Snowflake provider
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enhance Snowflake SQL API Hook with Retry Logic for Query Status Polling
5 participants
@astro-anand @SeiilkanM @potiuk @mik-laj @RNHTTR