[FG:InPlacePodVerticalScaling] Enable removing resource limits from containers #127143

hshiina · 2024-09-05T11:56:18Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR fixes handling cases where resources limits are max at InPlacePodVerticalScaling.

When one of containers in a pod has no limits for a resource, no limits will be configured in the cgroup of the pod. This fix allows resizing the resource in a container that has limits when another container in the pod does not have limits for the resource. In doPodResizeAction(), this fix allows a container to be resized even if the pod resources are unlimited (the values in the pod resource is nil).

This PR also enables removing resource limits from containers by following changes:

Update pod cgroup resources to max
Update container resources to max with CRI API UpdateContainerResource().
Display resource limits in container status while limits are being removed.

Which issue(s) this PR fixes:

Fixes #128675

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Enabled removing resource limits and requests at InPlacePodVerticalScaling.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2024-09-05T11:56:28Z

Hi @hshiina. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hshiina · 2024-09-05T12:13:09Z

/kind feature

kannon92 · 2024-09-11T17:04:35Z

/ok-to-test

hshiina · 2024-09-27T20:08:17Z

/test pull-kubernetes-node-kubelet-serial-podresize

tallclair · 2024-10-14T22:10:36Z

/assign

natasha41575 · 2025-04-30T19:45:34Z

pkg/kubelet/kubelet_pods.go

-				if cStatus.Resources.CPULimit.MilliValue() > cm.MinMilliCPULimit ||
-					resources.Limits.Cpu().MilliValue() > cm.MinMilliCPULimit {
-					resources.Limits[v1.ResourceCPU] = cStatus.Resources.CPULimit.DeepCopy()
+		if cStatus.Resources != nil && cStatus.Resources.CPULimit != nil {


nit / optional / preexisting (take it or leave it):

this code looks like you are doing (almost) the same thing 3 times in a row, once for cpu limits, then once for memory limits, then once for cpu requests... makes me feel like it can / should be put into a helper (though admittedly this may not be as straightforward as I am imagining in my head)

Though we should do so, fixing and refactoring at the same time could make review difficult. So, I don't introduce a helper in this PR.

natasha41575 · 2025-04-30T19:47:43Z

pkg/kubelet/kubelet_pods.go

 			}
+		} else if resources.Requests != nil && !resources.Requests.Cpu().IsZero() {
+			// Only if requests is in the allocation, restore limits to container sutatus from old status.


Suggested change

// Only if requests is in the allocation, restore limits to container sutatus from old status.

// Only if requests is in the allocation, restore limits to container status from old status.

I fixed. Thanks!

natasha41575 · 2025-04-30T20:02:08Z

pkg/kubelet/kuberuntime/kuberuntime_manager.go

+			value := maxValue
+			podResources.Memory = &value


nit

Suggested change

value := maxValue

podResources.Memory = &value

podResources.Memory = ptr.To(maxValue)

(import "k8s.io/utils/ptr")

same in other places

Done. Thanks.

natasha41575 · 2025-04-30T20:05:34Z

pkg/kubelet/kuberuntime/kuberuntime_container_linux.go

+		return nil
+	}
+	if m.cpuCFSQuota && containerResources.Linux.CpuQuota == 0 {
+		containerResources.Linux.CpuQuota = -1


Is this value -1 for the same reason that the maxValue const you added to pkg/kubelet/kuberuntime/kuberuntime_manager.go is -1?

If so, consider putting the maxValue const somewhere that it can be shared in both places

These are slightly different. This -1 is passed to the runtime for updating a container resource while the other is passed to libcontainer/cgroups for updating pod cgroups. I added a few comments.

natasha41575 · 2025-04-30T20:07:17Z

pkg/kubelet/kuberuntime/kuberuntime_manager.go

 	if podResources == nil {
 		klog.ErrorS(nil, "Unable to get resource configuration", "pod", pod.Name)
 		result.Fail(fmt.Errorf("unable to get resource configuration processing resize for pod %s", pod.Name))
 		return
 	}
+
+	const maxValue = int64(-1)


I see in your comment #127143 (comment) that this doesn't appear to be documented anywhere, maybe it'll be helpful to leave a comment here with this context so that in the future we can remember where this constant came from

I added a comment.

natasha41575 · 2025-04-30T20:07:55Z

test/e2e/framework/pod/resize.go

-			CPU     string `json:"cpu,omitempty"`
-			Memory  string `json:"memory,omitempty"`
-			EphStor string `json:"ephemeral-storage,omitempty"`
+			CPU     *string `json:"cpu"`


why do we need to change this?

I would like to allow null in a patch for removing requests and limits.

k8s-ci-robot · 2025-05-02T08:58:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hshiina
Once this PR has been reviewed and has the lgtm label, please assign smarterclayton for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hshiina

Thank you for your review.

hshiina · 2025-05-02T09:01:52Z

pkg/kubelet/kubelet_pods.go

-				if cStatus.Resources.CPULimit.MilliValue() > cm.MinMilliCPULimit ||
-					resources.Limits.Cpu().MilliValue() > cm.MinMilliCPULimit {
-					resources.Limits[v1.ResourceCPU] = cStatus.Resources.CPULimit.DeepCopy()
+		if cStatus.Resources != nil && cStatus.Resources.CPULimit != nil {


Though we should do so, fixing and refactoring at the same time could make review difficult. So, I don't introduce a helper in this PR.

hshiina · 2025-05-02T09:05:00Z

pkg/kubelet/kuberuntime/kuberuntime_container_linux.go

+		return nil
+	}
+	if m.cpuCFSQuota && containerResources.Linux.CpuQuota == 0 {
+		containerResources.Linux.CpuQuota = -1


These are slightly different. This -1 is passed to the runtime for updating a container resource while the other is passed to libcontainer/cgroups for updating pod cgroups. I added a few comments.

hshiina · 2025-05-02T09:05:49Z

pkg/kubelet/kuberuntime/kuberuntime_manager.go

 	if podResources == nil {
 		klog.ErrorS(nil, "Unable to get resource configuration", "pod", pod.Name)
 		result.Fail(fmt.Errorf("unable to get resource configuration processing resize for pod %s", pod.Name))
 		return
 	}
+
+	const maxValue = int64(-1)


I added a comment.

hshiina · 2025-05-02T09:06:02Z

pkg/kubelet/kuberuntime/kuberuntime_manager.go

+			value := maxValue
+			podResources.Memory = &value


Done. Thanks.

aojea · 2025-05-02T12:14:10Z

pkg/apis/core/validation/validation.go

-
-	return false
-}
-


this is changing a behavior in a graduated beta api, should you not first amend this change to the corresponding KEP instead of trying to get the change directly? or is this PR just asking for feedback?

oh, I See @tallclair already commented here #127143 (comment) ,

natasha41575

/lgtm

k8s-ci-robot · 2025-05-14T19:09:13Z

LGTM label has been added.

Git tree hash: b9a73fb753a3b57d866fa5738a6c2e463833ffd2

tallclair · 2025-05-16T22:36:26Z

@hshiina want to try and get this merged in v1.34?

hshiina · 2025-05-19T15:50:30Z

@hshiina want to try and get this merged in v1.34?

Yes, I would like to complete this work if possible.

This fix enables removing resource limits and requests from containers by following changes: - Update pod cgroup resources to max - Update container resources to max with CRI API UpdateContainerResource(). - Display resource limits in container status while limits are being removed.

k8s-ci-robot · 2025-05-25T17:29:05Z

New changes are detected. LGTM label has been removed.

k8s-ci-robot · 2025-05-25T18:30:29Z

@hshiina: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-unit-windows-master	`5052f25`	link	false	`/test pull-kubernetes-unit-windows-master`
pull-kubernetes-e2e-kind	`5052f25`	link	true	`/test pull-kubernetes-e2e-kind`
pull-kubernetes-e2e-gce	`5052f25`	link	true	`/test pull-kubernetes-e2e-gce`
pull-kubernetes-e2e-capz-windows-master	`5052f25`	link	false	`/test pull-kubernetes-e2e-capz-windows-master`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

tico88612 · 2025-06-30T14:07:59Z

Hello @hshiina @tallclair @natasha41575
This PR has not been updated for 1 month, so I'd like to check the status. If there's anything we can do, please let us know.
The code freeze is starting 02:00 UTC Friday 25th July 2025 (about 3 weeks from now). Please make sure the PR has both lgtm and approved labels before the code freeze. Thanks!

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Sep 5, 2024

k8s-ci-robot requested review from bart0sh and bobbypage September 5, 2024 11:56

k8s-ci-robot added area/kubelet area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 5, 2024

hshiina changed the title ~~[InPlacePodVerticalScaling~~ [WIP][InPlacePodVerticalScaling] Enable removing resource limits from containers Sep 5, 2024

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 5, 2024

hshiina changed the title ~~[WIP][InPlacePodVerticalScaling] Enable removing resource limits from containers~~ [WIP][FG:InPlacePodVerticalScaling] Enable removing resource limits from containers Sep 5, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 11, 2024

hshiina force-pushed the remove-limits branch from f5c6754 to 54bc5fd Compare September 27, 2024 19:03

k8s-ci-robot modified the milestones: v1.33, v1.34 Mar 21, 2025

natasha41575 reviewed Apr 30, 2025

View reviewed changes

hshiina force-pushed the remove-limits branch from b3f30f7 to cd4e0cf Compare May 2, 2025 08:58

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 2, 2025

hshiina commented May 2, 2025

View reviewed changes

hshiina force-pushed the remove-limits branch from cd4e0cf to 2955818 Compare May 2, 2025 09:11

aojea reviewed May 2, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 7, 2025

natasha41575 reviewed May 14, 2025

View reviewed changes

k8s-ci-robot assigned natasha41575 May 14, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 14, 2025

Rajalakshmi-Girish moved this from Tracked to Pending inclusion in [sig-release] Bug Triage May 21, 2025

natasha41575 mentioned this pull request May 23, 2025

KEP 1287: Instrumentation for in-place pod resize kubernetes/enhancements#5340

Merged

hshiina added 3 commits May 25, 2025 19:16

api: accept removing requsets & limits for Burstable pods

46d915f

Address feedback

5052f25

hshiina force-pushed the remove-limits branch from 2955818 to 5052f25 Compare May 25, 2025 17:29

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2025

k8s-ci-robot requested review from natasha41575 and tallclair May 25, 2025 17:29

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2025

tico88612 moved this from Pending inclusion to Tracked in [sig-release] Bug Triage Jun 30, 2025

	// Only if requests is in the allocation, restore limits to container sutatus from old status.
	// Only if requests is in the allocation, restore limits to container status from old status.

	value := maxValue
	podResources.Memory = &value
	podResources.Memory = ptr.To(maxValue)

[FG:InPlacePodVerticalScaling] Enable removing resource limits from containers #127143

Are you sure you want to change the base?

[FG:InPlacePodVerticalScaling] Enable removing resource limits from containers #127143

Uh oh!

Conversation

hshiina commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Sep 5, 2024

Uh oh!

hshiina commented Sep 5, 2024

Uh oh!

kannon92 commented Sep 11, 2024

Uh oh!

hshiina commented Sep 27, 2024

Uh oh!

tallclair commented Oct 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natasha41575 Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 2, 2025

Uh oh!

hshiina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natasha41575 left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 14, 2025

Uh oh!

tallclair commented May 16, 2025

Uh oh!

hshiina commented May 19, 2025

Uh oh!

k8s-ci-robot commented May 25, 2025

Uh oh!

k8s-ci-robot commented May 25, 2025

Uh oh!

hshiina commented Sep 5, 2024 •

edited

Loading

natasha41575 Apr 30, 2025 •

edited

Loading