Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm: adjust kubeadm skew policy for upgrades #120825

Merged
merged 2 commits into from
Oct 12, 2023
Merged

kubeadm: adjust kubeadm skew policy for upgrades #120825

merged 2 commits into from
Oct 12, 2023

Conversation

pacoxu
Copy link
Member

@pacoxu pacoxu commented Sep 22, 2023 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

xref kubernetes/kubeadm#2924

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kubeadm: allow deploying a kubelet that is 3 versions older than the version of kubeadm (N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands "init", "join" and "upgrade". Note that if the kubeadm user applies a control plane version that is older than the kubeadm version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2.

The KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/3935-oldest-node-newest-control-plane

@k8s-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 22, 2023
@pacoxu pacoxu marked this pull request as ready for review October 7, 2023 08:25
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 7, 2023
@pacoxu
Copy link
Member Author

pacoxu commented Oct 7, 2023 •
edited

I tested with 1.28 control-plane node with 1.26 kubelet node.

  • upgrade node works well.

I need to double check later.

@chendave
Copy link
Member

chendave commented Oct 7, 2023

/assign

chendave
chendave reviewed Oct 8, 2023
View reviewed changes
Copy link
Member

@chendave chendave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

but some unit should be fixed as well.

/assign @neolit123
for review and approval

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 8, 2023
@pacoxu
Copy link
Member Author

pacoxu commented Oct 8, 2023

/priority important-soon
/triage accepted

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 8, 2023
neolit123
neolit123 reviewed Oct 9, 2023
View reviewed changes
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/hold
for the above question

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Oct 9, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 42c5c65dfb0bdddc6055761d5e2f88c860aaa84f

@pacoxu
Copy link
Member Author

pacoxu commented Oct 9, 2023

I tested with 1.28 control-plane node with 1.26 kubelet node.

  • upgrade node works well.

I need to double check later.

we need the new skew to work on init and join without warnings too. does the KubeletVersion preflight check tolerates this?

After this change, the KubeletVersion check will use MinimumKubeletVersion = getSkewedKubernetesVersion(-3).

kubernetes/cmd/kubeadm/app/preflight/checks.go

Lines 610 to 612 in 0786fcc

if kubever.minKubeletVersion == nil {
kubever.minKubeletVersion = kubeadmconstants.MinimumKubeletVersion
}

@neolit123
Copy link
Member

we need the new skew to work on init and join without warnings too. does the KubeletVersion preflight check tolerates this?

After this change, the KubeletVersion check will use MinimumKubeletVersion = getSkewedKubernetesVersion(-3).

kubernetes/cmd/kubeadm/app/preflight/checks.go

Lines 610 to 612 in 0786fcc

if kubever.minKubeletVersion == nil {
kubever.minKubeletVersion = kubeadmconstants.MinimumKubeletVersion
}

but the skew becomes N-3 of the kubeadm version, while the sig architecture change is N-3 of the control plane version.

the current skew is documented here:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#version-skew-policy

if the user uses kubeadm 1.28, and applies a control-plane N-1 1.27, then if they use the N-3 kubelet skew it will be 1.25, but the skew between CP and kubelet will be N-2.

@neolit123
Copy link
Member

but the skew becomes N-3 of the kubeadm version, while the sig architecture change is N-3 of the control plane version.

i think this is still OK to do like so, i.e. we support the N-3 if the user doesn't use a custom control plane version.
does that make sense to you?

Fixes kubernetes/kubeadm#2924

this should not close the k/kubeadm ticket because we need e2e and docs changes too.

@neolit123
Copy link
Member

proposing release note:

kubeadm: allow deploying a kubelet that is 3 versions older than the version of kubeadm (N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands "init", "join" and "upgrade". Note that if the kubeadm user applies a control plane version that is older than the kubeadm version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2.

please double check my wording.

@pacoxu
Copy link
Member Author

pacoxu commented Oct 11, 2023

but the skew becomes N-3 of the kubeadm version, while the sig architecture change is N-3 of the control plane version.

I think this is still OK to do like so, i.e. we support the N-3 if the user doesn't use a custom control plane version.
does that make sense to you?

It sounds OK as well. Use kubeadm to upgrade to n-1 for the downgrade in most cases.

kubeadm can only upgrade to the kubeadm version or n-1 control plane.

  • and kubelet n-3.

pacoxu
pacoxu commented Oct 11, 2023
View reviewed changes
cmd/kubeadm/app/constants/constants.go Outdated
@@ -453,7 +453,8 @@ var (
MinimumControlPlaneVersion = getSkewedKubernetesVersion(-1)

// MinimumKubeletVersion specifies the minimum version of kubelet which kubeadm supports
MinimumKubeletVersion = getSkewedKubernetesVersion(-1)
// Refer to https://kubernetes.io/releases/version-skew-policy/#kubelet-1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add some comments that kubelet version can be N-3 of the kubeadm version, as control-plane is n or n-1, this can meet N-3 of the control plane version.

// Allows the kubelet version to be N-3 of the kubeadm version, provided that the control plane version is either N or N-1, meeting the maximum N-3 requirement for the control plane version.

Copy link
Member

@neolit123 neolit123 Oct 11, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually i think we should drop this comment and just leave the old comment (line 455):

// Refer to https://kubernetes.io/releases/version-skew-policy/#kubelet-1

the constants.go is not a place where users should read what we have. it's for us / developers to manage how kubeadm works.

if they (users) visit the k8s.io website they can find the kubeadm and k8s skews.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we update it to https://kubernetes.io/releases/version-skew-policy/?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it's ok to not link to docs. these comments are for developers to understand.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 12, 2023
neolit123
neolit123 reviewed Oct 12, 2023
View reviewed changes
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 12, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 3f5df90ed307e59775cfed002a1a8266c71eda37

SataQiu
SataQiu reviewed Oct 12, 2023
View reviewed changes
Copy link
Member

@SataQiu SataQiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pacoxu, SataQiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@neolit123
Copy link
Member

/hold cancel

@pacoxu
Copy link
Member Author

pacoxu commented Oct 12, 2023

ci bot hangs

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 12, 2023
@k8s-ci-robot k8s-ci-robot merged commit 801932c into kubernetes:master Oct 12, 2023
14 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Oct 12, 2023
@neolit123 neolit123 mentioned this pull request Nov 13, 2023
Closed
2 tasks
@kfox1111 kfox1111 mentioned this pull request Jan 8, 2024
Closed
@xmudrii xmudrii mentioned this pull request Jan 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@neolit123 neolit123 neolit123 left review comments

@SataQiu SataQiu SataQiu left review comments

@chendave chendave Awaiting requested review from chendave

Assignees

@neolit123 neolit123

@chendave chendave

@SataQiu SataQiu

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubeadm cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.29
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@pacoxu @k8s-ci-robot @chendave @neolit123 @SataQiu