Skip to content

Handle exceptions when fetching status in GlueJobHook #52262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 26, 2025
Merged

Handle exceptions when fetching status in GlueJobHook #52262

merged 2 commits into from
Jun 26, 2025
+204 −6

Conversation

AryanK1511
Copy link
Contributor

@AryanK1511 AryanK1511 commented Jun 25, 2025 •
edited by eladkal
Loading

fixes: #52152 by adding robust error handling and a retry mechanism to the get_job_state and async_get_job_state methods in the AWS Glue Provider for Airflow.

  • Implemented a retry strategy using tenacity for both methods, inspired by the approach in Snowflake SQL API hook Retry Logic #51463. Now, transient AWS API failures are retried up to 5 times before ultimately raising an exception if all attempts fail.
  • Added unit tests that mock various AWS API responses based on the official docs.
  • Test coverage includes:
    • Successful retries after transient failures
    • Failure after 5 attempts (ensuring the error is properly raised)

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 25, 2025
@eladkal eladkal requested a review from vincbeck June 25, 2025 17:31
@AryanK1511
Copy link
Contributor Author

The failing test seems flaky to me. Can someone try rerunning it?

@rawwar rawwar self-requested a review June 26, 2025 04:12
@AryanK1511 AryanK1511 force-pushed the aryank1511/issue-52152 branch from 7329c8f to f41902e Compare June 26, 2025 07:34
@vincbeck vincbeck merged commit 63b2207 into apache:main Jun 26, 2025
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vincbeck vincbeck vincbeck approved these changes

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@rawwar rawwar Awaiting requested review from rawwar

Assignees
No one assigned
Labels
area:providers provider:amazon AWS/Amazon - related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

GlueJobHook.get_job_state doesn't handle exceptions when fetching status
2 participants
@AryanK1511 @vincbeck