-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DRA: improve handling of completed pods #118817
Conversation
Skipping CI for Draft Pull Request. |
/remove-sig api-machinery |
/remove-sig api-machinery |
LGTM label has been added. Git tree hash: 85252cbb4a15bba606dfbb5cfbec32323675936a
|
RBAC change lgtm, once the other packages have approval I can ack |
pkg/api/v1/pod/util.go
Outdated
@@ -313,6 +313,13 @@ func IsPodTerminal(pod *v1.Pod) bool { | |||
return IsPodPhaseTerminal(pod.Status.Phase) | |||
} | |||
|
|||
// IsPodDone returns true if it is certain that none of the containers are running and never will run. | |||
func IsPodDone(pod *v1.Pod) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the distinction between IsPodDone and IsPodTerminal is going to be pretty confusing for callers... I expect people to choose incorrectly... when should someone use IsPodTerminal instead of IsPodDone?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure. Perhaps all users of IsPodTerminal
should also check for the additional condition in IsPodDone
? Then we don't need two functions. Let's see... there are exactly two calls outside of tests:
kubernetes/pkg/controller/job/job_controller.go
Line 1008 in 51429cb
if podutil.IsPodTerminal(pod) || considerPodFailed || finishedCond != nil || job.DeletionTimestamp != nil { kubernetes/pkg/controller/util/endpoint/controller_utils.go
Lines 92 to 112 in 51429cb
// ShouldPodBeInEndpoints returns true if a specified pod should be in an // Endpoints or EndpointSlice resource. Terminating pods are only included if // includeTerminating is true. func ShouldPodBeInEndpoints(pod *v1.Pod, includeTerminating bool) bool { // "Terminal" describes when a Pod is complete (in a succeeded or failed phase). // This is distinct from the "Terminating" condition which represents when a Pod // is being terminated (metadata.deletionTimestamp is non nil). if podutil.IsPodTerminal(pod) { return false } if len(pod.Status.PodIP) == 0 && len(pod.Status.PodIPs) == 0 { return false } if !includeTerminating && pod.DeletionTimestamp != nil { return false } return true }
The first looks like it would be okay with the semantic from IsPodDone
. The second has explicit handling of DeletionTimestamp
, so it's probably not okay to change it.
Let's defer this. I'll make IsPodDone
local to pkg/controller/resourceclaim/controller.go
for now and unless someone cares enough, will probably simply keep it there.
/cc @alculquicondor - you suggested moving this to pkg/api/v1/pod/util.go
earlier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@liggitt : okay to move ahead without the pkg/api/v1/pod change?
If yes, then please approve the auth change.
This covers pods that get deleted before running and will be used more than once soon.
Adding logging to event handlers makes it more obvious why (or why not) claims and pods need to be processed.
When a pod is known to never run (again), the reservation for it also can be removed. This is relevant in particular for the job controller.
When a pod is done, but not getting removed yet for while, then a claim that got generated for that pod can be deleted already. This then also triggers deallocation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
// deleted. Normally the garbage collector does that, but the | ||
// pod itself might not get deleted for a while. | ||
podName, podUID := owningPod(claim) | ||
if podName != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit, but if you reverse the if condition, you can make the code less complicated with:
if len(podName) == 0 {
logger.V(5).Info("claim not generated for a pod", "claim", klog.KObj(claim))
return nil
}
// the rest goes here
this allows the code be less indented, which is much more readable, and follows the general guidelines to "fail fast" which roughly the idea here 😉
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this particular case I prefer the current approach because it's not really a failure that gets checked for but rather a "do I need to do this thing here, if not, continue". If we ever add other things that might need to be done for a claim, then they would go before the current "return nil" at the end. That wouldn't work if some code in the middle returns.
OTOH, perhaps that's a reason to split up the function into smaller parts. Well, not now, okay?
LGTM label has been added. Git tree hash: d80b71b05f7ef270995a9c58257d21307fd5219b
|
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: byako, liggitt, pohly, soltysh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/triage accepted |
What type of PR is this?
/kind feature
What this PR does / why we need it:
When a pod is done or will never run because it got deleted before scheduling, then it doesn't need resources anymore. The reservation for the pod can be removed in all cases, which makes the claim usable for other pods. I addition, the claim itself can be deleted if it was generated specifically for the pod.
Which issue(s) this PR fixes:
Fixes #113890
Special notes for your reviewer:
Moving the "pod is deleted and not scheduled" check into podutils was suggested during the initial DRA code review and just hadn't been done at that time. Now the DRA controller itself needs it more than once, so it makes sense to do it now.
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: