Fix handling of subprocess error handling in s3_file_transform and gcs (

#9106) As outlined in Issue 9104 the python subprocess return code can be less than 0. The previous version only captures errors of the subprocess itself, not the negative error codes, as cause by host SIGKILL, SIGHUP, ... see https://docs.python.org/3/library/subprocess.html#subprocess. CompletedProcess.returncode We now treat all non-zero returncodes as a script failure.
apache · Jun 2, 2020 · 17adcea · 17adcea
1 parent 9e0ccde
commit 17adcea
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 2 deletions.
diff --git a/airflow/providers/amazon/aws/operators/s3_file_transform.py b/airflow/providers/amazon/aws/operators/s3_file_transform.py
@@ -154,7 +154,7 @@ def execute(self, context):
 
  process.wait()
 
- if process.returncode > 0:
+ if process.returncode:
  raise AirflowException(
  "Transform script failed: {0}".format(process.returncode)
  )

diff --git a/airflow/providers/google/cloud/operators/gcs.py b/airflow/providers/google/cloud/operators/gcs.py
@@ -609,7 +609,7 @@ def execute(self, context: Dict):
  self.log.info(line.decode(self.output_encoding).rstrip())
 
  process.wait()
- if process.returncode > 0:
+ if process.returncode:
  raise AirflowException(
  "Transform script failed: {0}".format(process.returncode)
  )