[PodLevelResources] Pod Level Resources Eviction Manager #132277

KevinTMtz · 2025-06-12T23:38:34Z

What type of PR is this?

What this PR does / why we need it:

This PR implements Pod Level Resources Eviction Manager that require following changes:

Use pod level resources for eviction manager
Unit tests for pod level resources eviction manager
E2E tests for pod level resources eviction manager
E2E tests for pod level resources Kubelet Preemption

Which issue(s) this PR is related to:

Fixes #132448

Special notes for your reviewer:

Does this PR introduce a user-facing change?

- No, changes underlying logic for Eviction Manager helper functions

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2025-06-12T23:38:43Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-06-12T23:38:43Z

Hi @KevinTMtz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

KevinTMtz · 2025-06-13T00:02:13Z

/assign @ndixita

ndixita · 2025-06-13T20:33:23Z

/ok-to-test

toVersus · 2025-06-15T09:43:02Z

pkg/kubelet/eviction/helpers.go

+
+		switch resourceToReclaim {
+		case v1.ResourceMemory:
+			podUsage = memoryUsage(podStats.Memory)


There may be cases where stats cannot be collected via the Kubelet Summary API, so it might be better to perform a nil check for podStats.Memory here, just like with container-level resource requests.

Done, thank you

toVersus · 2025-06-15T09:45:33Z

pkg/kubelet/eviction/helpers.go

+			podUsage = resource.NewQuantity(int64(*podStats.EphemeralStorage.UsedBytes), resource.BinarySI)
+		}
+
+		message += fmt.Sprintf("Pod %s was using %s, request is %s, has larger consumption of %v. ", pod.Name, podUsage.String(), podRequest.String(), resourceToReclaim)


nit: It might be better to extract the format of this message into a variable (e.g. podMessageFmt), similar to containerMessageFmt.

Done, thank you

toVersus · 2025-06-15T12:49:21Z

pkg/kubelet/eviction/helpers.go

+			podUsage = resource.NewQuantity(int64(*podStats.EphemeralStorage.UsedBytes), resource.BinarySI)
+		}
+
+		message += fmt.Sprintf("Pod %s was using %s, request is %s, has larger consumption of %v. ", pod.Name, podUsage.String(), podRequest.String(), resourceToReclaim)


Just a thought. Consider a Pod that uses pod-level resources and contains three containers.

apiVersion: v1 kind: Pod metadata: name: evicted spec: resources: requests: memory: 500Mi initContainers: - name: sidecar1 image: registry.k8s.io/e2e-test-images/agnhost:2.55 command: ["sleep", "infinity"] - name: sidecar2 image: registry.k8s.io/e2e-test-images/agnhost:2.55 command: ["sleep", "infinity"] containers: - name: regular image: registry.k8s.io/e2e-test-images/agnhost:2.55 command: ["sleep", "infinity"] resources: requests: memory: 300Mi

If an eviction occurs due to memory usage from sidecar1 or sidecar2, I feel like the current event and annotation alone doesn’t clearly indicate which container actually caused it. If pod-level resources are specified and none of the containers that have memory requests exceed their requested memory, it might be helpful to record the memory usage of the container without a memory request that used the most memory. That way, we can at least get a hint about which container likely triggered the eviction. Though, maybe this is overkill and not really necessary.

After getting the pod level resource stats, the function still continues and iterates all containers (like it used to do), so the user would get something like this:

Warning Evicted 49s kubelet The node was low on resource: memory. Threshold quantity: 80996576029, available: 75619264Ki. Pod stressor-sidecar-pod was using 12981880Ki, request is 1G, has larger consumption of memory. Container sidecar-stressor-container was using 10141636Ki, request is 0, has larger consumption of memory. Normal Killing 49s kubelet Stopping container sidecar-stressor-container

It shows the comparison between the whole pod usage against its pod level request, along the usage and request of each container.

Thanks! I didn't take into account that when requests isn't specified, it always shows the actual usage along with a message saying it exceeds zero.

ndixita · 2025-06-16T23:04:22Z

pkg/api/v1/resource/helpers.go

@@ -39,16 +42,23 @@ func GetResourceRequestQuantity(pod *v1.Pod, resourceName v1.ResourceName) resou
 		requestQuantity = resource.Quantity{Format: resource.DecimalSI}
 	}

-	for _, container := range pod.Spec.Containers {
-		if rQuantity, ok := container.Resources.Requests[resourceName]; ok {
+	// Supported pod level resources will be used instead of container level ones when available


GetResourceRequestQuantity is called in

kubernetes/pkg/api/v1/resource/helpers.go

Line 73 in 74210dd

requestQuantity := GetResourceRequestQuantity(pod, resource)

which is used in premption and allocation manager related files. Can we please add the e2e coverage for preemption logic as well along with eviction manager related e2e...

Done, thank you.

k8s-ci-robot · 2025-06-18T02:27:53Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: KevinTMtz
Once this PR has been reviewed and has the lgtm label, please assign mrunalp, thockin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

toVersus · 2025-07-01T00:24:22Z

test/e2e_node/critical_pod_test.go

@@ -147,6 +147,66 @@ var _ = SIGDescribe("CriticalPod", framework.WithSerial(), framework.WithDisrupt
 	})
 })

+var _ = SIGDescribe("CriticalPodWithPodLevelResources", framework.WithSerial(), framework.WithDisruptive(), feature.PodLevelResources, func() {


Since this is the first addition of Node E2E tests related to Pod Level Resources feature, kubernetes/test-infra#35061 seems necessary to pass presubmit job (pull-kubernetes-node-e2e-containerd-alpha-features).

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jun 12, 2025

k8s-ci-robot requested review from feiskyer and matthyx June 12, 2025 23:38

k8s-ci-robot added area/kubelet area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jun 12, 2025

github-project-automation bot moved this to Triage in SIG Node CI/Test Board Jun 12, 2025

github-project-automation bot added this to SIG Node: code and documentation PRs and SIG Node CI/Test Board Jun 12, 2025

k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jun 12, 2025

github-project-automation bot moved this to Triage in SIG Node: code and documentation PRs Jun 12, 2025

KevinTMtz force-pushed the pod-level-resources-eviction-manager branch from f0a8d0f to 4a2fd75 Compare June 12, 2025 23:51

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 12, 2025

KevinTMtz changed the title ~~Pod level resources eviction manager~~ [PodLevelResources] Pod Level Resources Eviction Manager Jun 13, 2025

k8s-ci-robot assigned ndixita Jun 13, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jun 13, 2025

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 13, 2025

toVersus reviewed Jun 15, 2025

View reviewed changes

bart0sh moved this from Triage to Work in progress in SIG Node: code and documentation PRs Jun 15, 2025

Use pod level resources for eviction manager

dec7b9b

KevinTMtz force-pushed the pod-level-resources-eviction-manager branch from 4a2fd75 to dec7b9b Compare June 16, 2025 19:00

ndixita reviewed Jun 16, 2025

View reviewed changes

ndixita moved this from Triage to PRs Waiting on Author in SIG Node CI/Test Board Jun 17, 2025

Unit tests for pod level resources eviction manager

bab90c6

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 17, 2025

E2E tests for pod level resources eviction manager

cc0b91c

E2E tests for pod level resources Kubelet Preemption

ffd526f

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jun 20, 2025

toVersus mentioned this pull request Jul 1, 2025

[PodLevelResources] Add missing service feature gate for node e2e alpha job kubernetes/test-infra#35061

Open

toVersus reviewed Jul 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PodLevelResources] Pod Level Resources Eviction Manager #132277

[PodLevelResources] Pod Level Resources Eviction Manager #132277

KevinTMtz commented Jun 12, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

KevinTMtz commented Jun 13, 2025

Uh oh!

ndixita commented Jun 13, 2025

Uh oh!

toVersus Jun 15, 2025

Uh oh!

KevinTMtz Jun 16, 2025

Uh oh!

toVersus Jun 15, 2025

Uh oh!

KevinTMtz Jun 16, 2025

Uh oh!

toVersus Jun 15, 2025

Uh oh!

KevinTMtz Jun 18, 2025

Uh oh!

toVersus Jun 18, 2025

Uh oh!

ndixita Jun 16, 2025

Uh oh!

KevinTMtz Jun 30, 2025

Uh oh!

k8s-ci-robot commented Jun 18, 2025

Uh oh!

toVersus Jul 1, 2025

Uh oh!

Uh oh!

[PodLevelResources] Pod Level Resources Eviction Manager #132277

Are you sure you want to change the base?

[PodLevelResources] Pod Level Resources Eviction Manager #132277

Conversation

KevinTMtz commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR is related to:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

KevinTMtz commented Jun 13, 2025

Uh oh!

ndixita commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KevinTMtz commented Jun 12, 2025 •

edited

Loading