Fix pod and container level swap metrics for CRI #132065

yuanwang04 · 2025-06-03T02:46:59Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixes the missing pod-level and container-level swap metrics from the CRI provider.

Swap metrics are introduced in #118865. For the CRI provider, the swap data is only scraped from the listPodStatsPartiallyFromCRI function. However, the resource metrics are scraped from the ListPodCPUAndMemoryStats function. This PR added the similar code changes to the ListPodCPUAndMemoryStats function so that the swap metrics can be reported correctly.

Also added the swap related metrics to the e2e test to assert these metrics exist.

Which issue(s) this PR fixes:

Fixes #131915

Special notes for your reviewer:

Also added the swap metrics to the e2e test suite

Does this PR introduce a user-facing change?

Fixed the bug when swap related metrics were not available in `/metrics/resource` endpoint.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2025-06-03T02:47:08Z

Welcome @yuanwang04!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-06-03T02:47:08Z

Hi @yuanwang04. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ajaysundark · 2025-06-04T17:14:14Z

/cc

SergeyKanzhelev · 2025-06-04T17:15:27Z

/ok-to-test
/assign @haircommander
/priority important-soon
/triage accepted

ajaysundark · 2025-06-07T08:06:24Z

pkg/kubelet/stats/cri_stats_provider.go

Does it also need a addCRIPodSwapUsage method here?

@iholder101 I don't see the Swap stats are being populated from the reference to CRI's ListPodSandboxStats function. Do you remember if this is a limitation of the CRI that Swap related data have to be retrieved from the cadvisor / cgroup directly?

Hey there! Sorry for the late reply.

In #118865, I was adding a comment regarding that:

Special notes for your reviewer:

AFAICT, cadvisor is being used to fetch the metric data. If it is not available through cadvisor, it's possible to fallback into gathering CRI data. Unfortunately, it doesn't seem currently possible to do so (here) since memory usage is not part of the CRI-API. In follow-up work, I think it should be added as well, as I don't see a reason to why CRIs wouldn't be able to provide this information.

While this was true back then, a swapUsage struct is now part of the CRI-API.

Therefore:

Does it also need a addCRIPodSwapUsage method here?

@ajaysundark, I think so, yes.

Do you remember if this is a limitation of the CRI that Swap related data have to be retrieved from the cadvisor / cgroup directly?

Prior to my work, this wasn't part of the CRI-API. Now it is. I'm not aware of any further limitation.

@haircommander can you please confirm that CRIs are implementing the swapUsage API?

https://github.com/cri-o/cri-o/blob/155deb668c7b54a0bed35f2e091d4f3067c93208/internal/lib/stats/stats_server_linux.go#L328-L333 looks like CRI-O does, but to return those you'd need to turn on the CRI stats feature, so it doesn't OOTB

@haircommander Could you help me understand the status of the CRI stats provider. This feature still seems to be alpha: https://kubernetes.io/docs/reference/instrumentation/cri-pod-container-metrics/?

containerd doesn't seem to support swapUsage in stats yet. I raised a bug to follow-up on this: containerd/containerd#12026.

CRI stats provider is used by default in containerd right now. there are two different halves of it. the legacy "partial" CRI stats (where the remainder are populated by cadvisor) vs the new (2371) full CRI stats. it's possible swap stats are only populated in the "full" case

yuanwang04 · 2025-06-12T17:31:22Z

@haircommander @kannon92 Could you help to review this PR? Thanks! I think we might also want to backport this fix to at least 1.33 so Swap can be monitored appropriately, cc @ajaysundark

pkg/kubelet/stats/cri_stats_provider.go

iholder101 · 2025-06-15T09:15:43Z

However, the resource metrics are scraped from the ListPodCPUAndMemoryStats function. This PR added the similar code changes to the ListPodCPUAndMemoryStats function so that the swap metrics can be reported correctly.

For the record, originally, I wasn't sure if swap should be considered as "CPU and memory" stats.
I wasn't aware that the /metrics/resource/ endpoint relies solely on these.

If so, I think this is the right direction 👍
Thank you @yuanwang04!

iholder101 · 2025-06-15T09:29:48Z

@haircommander @kannon92 Could you help to review this PR? Thanks! I think we might also want to backport this fix to at least 1.33 so Swap can be monitored appropriately, cc @ajaysundark

I tend to think this should be possible, if we treat this as a bug fix.

haircommander · 2025-06-24T14:22:01Z

/lgtm

k8s-ci-robot · 2025-06-24T14:22:09Z

LGTM label has been added.

Git tree hash: 205b8efb109b34df96a8735c9728881234977da7

yujuhong · 2025-06-25T18:29:34Z

Need a release note

SergeyKanzhelev

/lgtm
/approve

k8s-ci-robot · 2025-06-25T20:04:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SergeyKanzhelev, yuanwang04

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/stats/OWNERS~~ [SergeyKanzhelev]
~~test/e2e_node/OWNERS~~ [SergeyKanzhelev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

SergeyKanzhelev · 2025-06-25T20:06:15Z

/release-note-edit

Fixed the bug when swap related metrics were not available in `/metrics/resource` endpoint.

yuanwang04 · 2025-06-25T22:16:29Z

/retest

Error is from resource_quota test, seems irrelevant from this change

 I0625 21:26:18.631640 67863 resource_quota.go:1615] Unexpected error: 
      <*errors.StatusError | 0xc00203bb80>: 
      pods "terminating-pod" is forbidden: exceeded quota: quota-terminating, requested: pods=1, used: pods=1, limited: pods=1
      {
          ErrStatus: 
              code: 403
              details:
                kind: pods
                name: terminating-pod
              message: 'pods "terminating-pod" is forbidden: exceeded quota: quota-terminating,
                requested: pods=1, used: pods=1, limited: pods=1'
              metadata: {}
              reason: Forbidden
              status: Failure,
      }
  [FAILED] in [It] - k8s.io/kubernetes/test/e2e/apimachinery/resource_quota.go:1615 @ 06/25/25 21:26:18.631

github-project-automation bot added this to SIG Node CI/Test Board Jun 3, 2025

k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jun 3, 2025

github-project-automation bot moved this to Triage in SIG Node CI/Test Board Jun 3, 2025

k8s-ci-robot requested review from kannon92 and krmayankk June 3, 2025 02:47

k8s-ci-robot assigned haircommander Jun 4, 2025

github-project-automation bot added this to SIG Node: code and documentation PRs Jun 4, 2025

k8s-ci-robot removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jun 4, 2025

github-project-automation bot moved this to Triage in SIG Node: code and documentation PRs Jun 4, 2025

SergeyKanzhelev moved this from Triage to Archive-it in SIG Node CI/Test Board Jun 4, 2025

ajaysundark reviewed Jun 7, 2025

View reviewed changes

haircommander reviewed Jun 12, 2025

View reviewed changes

pkg/kubelet/stats/cri_stats_provider.go Outdated Show resolved Hide resolved

Fix pod and container level swap metrics for CRI

c5f061e

yuanwang04 force-pushed the SwapMetrics branch from 7076008 to c5f061e Compare June 23, 2025 17:57

yuanwang04 requested a review from haircommander June 23, 2025 17:57

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 24, 2025

SergeyKanzhelev approved these changes Jun 25, 2025

View reviewed changes

k8s-ci-robot assigned SergeyKanzhelev Jun 25, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 25, 2025

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jun 25, 2025

k8s-ci-robot merged commit 1e59323 into kubernetes:master Jun 25, 2025
14 checks passed

k8s-ci-robot added this to the v1.34 milestone Jun 25, 2025

github-project-automation bot moved this from Triage to Done in SIG Node: code and documentation PRs Jun 25, 2025

github-project-automation bot moved this from Archive-it to Done in SIG Node CI/Test Board Jun 25, 2025

Fix pod and container level swap metrics for CRI #132065

Fix pod and container level swap metrics for CRI #132065

Conversation

yuanwang04 commented Jun 3, 2025 • edited by k8s-ci-robot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Jun 3, 2025

Uh oh!

k8s-ci-robot commented Jun 3, 2025

Uh oh!

ajaysundark commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergeyKanzhelev commented Jun 4, 2025

Uh oh!

ajaysundark Jun 7, 2025

Choose a reason for hiding this comment

Uh oh!

yuanwang04 Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

iholder101 Jun 15, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

ajaysundark Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

haircommander Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

yuanwang04 commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

iholder101 commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iholder101 commented Jun 15, 2025

Uh oh!

haircommander commented Jun 24, 2025

Uh oh!

k8s-ci-robot commented Jun 24, 2025

Uh oh!

yujuhong commented Jun 25, 2025

Uh oh!

SergeyKanzhelev left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 25, 2025

Uh oh!

SergeyKanzhelev commented Jun 25, 2025

Uh oh!

yuanwang04 commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

yuanwang04 commented Jun 3, 2025 •

edited by k8s-ci-robot

Loading

ajaysundark commented Jun 4, 2025 •

edited

Loading

yuanwang04 commented Jun 12, 2025 •

edited

Loading

iholder101 commented Jun 15, 2025 •

edited

Loading