Fix deferrable mode for SparkKubernetesOperator #51956
Merged
+60
−16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sunank200
reviewed
Jun 20, 2025
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
Outdated
Show resolved
Hide resolved
db40f8e to
7666db9
Compare
@sunank200 I have updated the description and added a new test |
7666db9 to
359d7bd
Compare
359d7bd to
c453684
Compare
|
@potiuk Hi there! |
potiuk
approved these changes
Jun 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR I have fixed deferrable mode for
SparkKubernetesOperator.The error appears at the beginning of
execute_asyncwhen the operator tries to run thebuild_pod_request_objfunction. This function builds a pod object, but theSparkKubernetesOperatoroperator also creates this object in its execute method. And creating the second time this object produce the error:But
KubernetesPodOperatorproduces the error only for deferrable mode in theexecute_asyncmethod. Because forexecute_syncthe logic was updated to prevent this error in the initial PR aa25aff#diff-93707635f12195e7b94d14cf99675542592a28a6eb552895fb3ff37d42713740 forSparkKubernetesOperator.In the current PR I have equalized the logic for both modes.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.