Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: broadcast when pod is pushed back to activeQ directly in AddUnschedulableIfNotPresent #119784

Merged
merged 1 commit into from
Aug 9, 2023
Merged

fix: broadcast when pod is pushed back to activeQ directly in AddUnschedulableIfNotPresent #119784

merged 1 commit into from
Aug 9, 2023

Conversation

sanposhiho
Copy link
Member

@sanposhiho sanposhiho commented Aug 6, 2023 •
edited

What type of PR is this?

/kind bug

What this PR does / why we need it:

p.cond.Broadcast() when pod is pushed back to activeQ directly in AddUnschedulableIfNotPresent.
Otherwise, the Pod could stick until another Pod is pushed to activeQ. In other words, if the cluster is super small and only one Pod is unschedulable now, the Pod won't be scheduled forever.

The scenario facing this bug will be covered in test added in #119105

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

@kubernetes/sig-scheduling-leads I'd like to get this in v1.28. 🙏

Does this PR introduce a user-facing change?

When the cluster size is small and the scheduler doesn't get unscheduled Pods frequently, the scheduler doesn't try to reschedule Pods in some cases.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. labels Aug 6, 2023
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Sun Aug 6 04:30:20 UTC 2023.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 6, 2023
@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 6, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 6, 2023
@sanposhiho sanposhiho mentioned this pull request Aug 6, 2023
Merged
@sanposhiho
Copy link
Member Author

/retest

@alculquicondor
Copy link
Member

The test needs to be in this PR if you intend to add it to the milestone and/or cherry-pick.
Please add a release note.

Is this a regression?

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Aug 9, 2023
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 9, 2023
@sanposhiho
Copy link
Member Author

Added the test as suggested.

Is this a regression?

It's a regression by the queueing hint, I believe it's not happening in older versions that don't have the queueing hint change.
Because the older version's AddUnschedulableIfNotPresent always put Pods to backoffQ, never put to activeQ (ref), and then flushBackoffQCompleted puts the Pod to activeQ later if the Pod doesn't need to be in backoffQ. After introducing queueing hint, AddUnschedulableIfNotPresent gets a bit wiser, and can directly put Pods to activeQ. (and I forgot to put p.cond.Broadcast() there then)

So, no need to cherry-pick this to older versions, but need to /milestone to get this in 1.28.

@sanposhiho
Copy link
Member Author

/retest

@alculquicondor
Copy link
Member

/kind regression

@Huang-Wei can you add to the milestone?

@k8s-ci-robot k8s-ci-robot added the kind/regression Categorizes issue or PR as related to a regression from a prior release. label Aug 9, 2023
alculquicondor
alculquicondor reviewed Aug 9, 2023
View reviewed changes
Copy link
Member

@alculquicondor alculquicondor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Huang-Wei
Copy link
Member

/milestone v1.28

@k8s-ci-robot k8s-ci-robot added this to the v1.28 milestone Aug 9, 2023
@alculquicondor
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 9, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 83e049cf86a566854d74544400cf9b28b2926678

@Huang-Wei
Copy link
Member

/retest

1 similar comment
@alculquicondor
Copy link
Member

/retest

@k8s-ci-robot k8s-ci-robot merged commit 201f807 into kubernetes:master Aug 9, 2023
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alculquicondor alculquicondor alculquicondor left review comments

@chendave chendave Awaiting requested review from chendave

@damemi damemi Awaiting requested review from damemi

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
[sig-release] Bug Triage
Archived in project
Milestone
v1.28
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@sanposhiho @k8s-ci-robot @alculquicondor @Huang-Wei