Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRA: improve handling of completed pods #118817

Merged
merged 4 commits into from
Jul 6, 2023

Conversation

pohly
Copy link
Contributor

@pohly pohly commented Jun 22, 2023

What type of PR is this?

/kind feature

What this PR does / why we need it:

When a pod is done or will never run because it got deleted before scheduling, then it doesn't need resources anymore. The reservation for the pod can be removed in all cases, which makes the claim usable for other pods. I addition, the claim itself can be deleted if it was generated specifically for the pod.

Which issue(s) this PR fixes:

Fixes #113890

Special notes for your reviewer:

Moving the "pod is deleted and not scheduled" check into podutils was suggested during the initial DRA code review and just hadn't been done at that time. Now the DRA controller itself needs it more than once, so it makes sense to do it now.

Does this PR introduce a user-facing change?

When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/3063

@k8s-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jun 22, 2023
@k8s-ci-robot k8s-ci-robot added area/test sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 22, 2023
@pohly pohly marked this pull request as ready for review June 22, 2023 13:04
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 22, 2023
@jiahuif
Copy link
Member

jiahuif commented Jun 22, 2023

/remove-sig api-machinery

@k8s-ci-robot k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jun 22, 2023
@pacoxu pacoxu added this to Triage in SIG Node PR Triage Jun 24, 2023
@SergeyKanzhelev SergeyKanzhelev moved this from Triage to Archive-it in SIG Node CI/Test Board Jun 28, 2023
@k8s-ci-robot k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jun 29, 2023
@jiahuif
Copy link
Member

jiahuif commented Jun 29, 2023

/remove-sig api-machinery

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 1, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 85252cbb4a15bba606dfbb5cfbec32323675936a

@liggitt
Copy link
Member

liggitt commented Jul 5, 2023

RBAC change lgtm, once the other packages have approval I can ack

@@ -313,6 +313,13 @@ func IsPodTerminal(pod *v1.Pod) bool {
return IsPodPhaseTerminal(pod.Status.Phase)
}

// IsPodDone returns true if it is certain that none of the containers are running and never will run.
func IsPodDone(pod *v1.Pod) bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the distinction between IsPodDone and IsPodTerminal is going to be pretty confusing for callers... I expect people to choose incorrectly... when should someone use IsPodTerminal instead of IsPodDone?

Copy link
Contributor Author

@pohly pohly Jul 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure. Perhaps all users of IsPodTerminal should also check for the additional condition in IsPodDone? Then we don't need two functions. Let's see... there are exactly two calls outside of tests:

  • if podutil.IsPodTerminal(pod) || considerPodFailed || finishedCond != nil || job.DeletionTimestamp != nil {
  • // ShouldPodBeInEndpoints returns true if a specified pod should be in an
    // Endpoints or EndpointSlice resource. Terminating pods are only included if
    // includeTerminating is true.
    func ShouldPodBeInEndpoints(pod *v1.Pod, includeTerminating bool) bool {
    // "Terminal" describes when a Pod is complete (in a succeeded or failed phase).
    // This is distinct from the "Terminating" condition which represents when a Pod
    // is being terminated (metadata.deletionTimestamp is non nil).
    if podutil.IsPodTerminal(pod) {
    return false
    }
    if len(pod.Status.PodIP) == 0 && len(pod.Status.PodIPs) == 0 {
    return false
    }
    if !includeTerminating && pod.DeletionTimestamp != nil {
    return false
    }
    return true
    }

The first looks like it would be okay with the semantic from IsPodDone. The second has explicit handling of DeletionTimestamp, so it's probably not okay to change it.

Let's defer this. I'll make IsPodDone local to pkg/controller/resourceclaim/controller.go for now and unless someone cares enough, will probably simply keep it there.

/cc @alculquicondor - you suggested moving this to pkg/api/v1/pod/util.go earlier.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt : okay to move ahead without the pkg/api/v1/pod change?

If yes, then please approve the auth change.

pohly added 4 commits July 5, 2023 16:09
This covers pods that get deleted before running and will be used more than
once soon.
Adding logging to event handlers makes it more obvious why (or why not) claims
and pods need to be processed.
When a pod is known to never run (again), the reservation for it also can be
removed. This is relevant in particular for the job controller.
When a pod is done, but not getting removed yet for while, then a claim that
got generated for that pod can be deleted already. This then also triggers
deallocation.
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 5, 2023
Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

// deleted. Normally the garbage collector does that, but the
// pod itself might not get deleted for a while.
podName, podUID := owningPod(claim)
if podName != "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but if you reverse the if condition, you can make the code less complicated with:

if len(podName) == 0 {
     logger.V(5).Info("claim not generated for a pod", "claim", klog.KObj(claim))
     return nil
}
// the rest goes here

this allows the code be less indented, which is much more readable, and follows the general guidelines to "fail fast" which roughly the idea here 😉

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this particular case I prefer the current approach because it's not really a failure that gets checked for but rather a "do I need to do this thing here, if not, continue". If we ever add other things that might need to be done for a claim, then they would go before the current "return nil" at the end. That wouldn't work if some code in the middle returns.

OTOH, perhaps that's a reason to split up the function into smaller parts. Well, not now, okay?

SIG Node CI/Test Board automation moved this from Archive-it to PRs - Needs Approver Jul 5, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 5, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: d80b71b05f7ef270995a9c58257d21307fd5219b

@liggitt
Copy link
Member

liggitt commented Jul 6, 2023

/approve
for rbac change

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: byako, liggitt, pohly, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2023
@jiahuif
Copy link
Member

jiahuif commented Jul 6, 2023

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 6, 2023
@k8s-ci-robot k8s-ci-robot merged commit 6f9d1d3 into kubernetes:master Jul 6, 2023
15 checks passed
SIG Node CI/Test Board automation moved this from PRs - Needs Approver to Done Jul 6, 2023
SIG Node PR Triage automation moved this from Triage to Done Jul 6, 2023
@k8s-ci-robot k8s-ci-robot added this to the v1.28 milestone Jul 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Archived in project
Archived in project
Development

Successfully merging this pull request may close these issues.

dynamic resource allocation: delete generated ResourceClaims
7 participants