Skip to content

Fix handling of subprocess error handling in s3_file_transform and gcs #9106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 2, 2020
Merged

Fix handling of subprocess error handling in s3_file_transform and gcs #9106

merged 1 commit into from
Jun 2, 2020

Conversation

blackw1ng
Copy link
Contributor

@blackw1ng blackw1ng commented Jun 2, 2020 •
edited
Loading

As outlined in #9104 the python subprocess return code can be less than 0, which also indicates an error.

The previous version only captures errors ( >0 ) of the subprocess itself, not the negative error codes, as cause by host SIGKILL, SIGHUP, ...
see Python subprocess documentation

We now treat all non-zero return codes as a script failure - similar to what most other operators do.

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

@blackw1ng
Fix handling of subprocess error handling in s3_file_transform and gcs
74a0332 
As outlined in Issue 9104 the python subprocess return code can be less than 0.
The previous version only captures errors of the subprocess itself, not the negative error codes, as cause by host SIGKILL, SIGHUP, ... see https://docs.python.org/3/library/subprocess.html#subprocess.
CompletedProcess.returncode

We now treat all non-zero returncodes as a script failure.
@boring-cyborg boring-cyborg bot added provider:amazon AWS/Amazon - related issues provider:google Google (including GCP) related issues labels Jun 2, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 2, 2020

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://apache-airflow-slack.herokuapp.com/

@potiuk potiuk merged commit 17adcea into apache:master Jun 2, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 2, 2020

Awesome work, congrats on your first merged pull request!

@potiuk
Copy link
Member

potiuk commented Jun 2, 2020

THanks! @blackw1ng !

@blackw1ng
Copy link
Contributor Author

Glad to help. It took us a while to figure it out - hopefully saves someone else some pains.

@blackw1ng blackw1ng deleted the Issue9104_fix_subprocess_exit_code branch June 26, 2020 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potiuk potiuk potiuk approved these changes

Assignees
No one assigned
Labels
provider:amazon AWS/Amazon - related issues provider:google Google (including GCP) related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@blackw1ng @potiuk