[KEP-2400] kubectl top: add a --show-swap option #129458

iholder101 · 2025-01-02T16:58:17Z

What type of PR is this?

/kind feature

Background

A while ago, swap stats were added to summary and metric APIs as part of #118865. Although these stats are extremely valuable, they're mostly designed to be consumed by programs (like HPAs or metrics solutions), not by humans. This raised the request for a better way of understanding which nodes have swap, the swap capacity and usage, which pods consume swap space and how much, etc.

I've tried to address this need in two past attempts.
In the first one, #123963, I've tried to add swap as another ReasourceName, hence to display it as part of kubectl describe node's status. This attempt was eventually denied with the main concern being that adding swap as a ResourceName is confusing since it cannot be directly/explicitly consumed (since the swap limit is auto-calculated).

In the second one, #125278, I've tried to add a swap condition to nodes. This PR was also denied with the main concern being that a condition is not suited to report such data, and that a field is recommended in this case, although it wasn't easy thinking of a good field that will benefit the API.

Eventually we agreed to defer this to the future. However, while attempting to GA the swap feature there was a hard push-back against GA without addressing these needs.

Therefore, this is my third attempt to address it. My hope is that it will suffice the need of debugability and will pave the way for the upcoming GA.

What this PR does / why we need it:

This PR adds adds the --show-swap option to the kubectl top subcommands.

Behind the scenes, this purely relies on consuming the stats added in #118865, making them human-readable.

Example output:

    > kubectl top node --show-swap
    NAME     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)   SWAP CAPACITY(bytes)
    node01   500m         8%       2836Mi          60%         0Mi           0%        0Mi
    node02   260m         5%       2206Mi          47%         102Mi         10%       1023Mi

    > kubectl top pod -n kube-system --show-swap
    NAME                                      CPU(cores)   MEMORY(bytes)   SWAP(bytes)
    coredns-58d5bc5cdb-5nbk4                  2m           19Mi            0Mi
    coredns-58d5bc5cdb-jsh26                  3m           37Mi            0Mi
    etcd-node01                               51m          143Mi           0Mi
    kube-apiserver-node01                     98m          824Mi           0Mi
    kube-controller-manager-node01            20m          135Mi           0Mi
    kube-proxy-ffgs2                          1m           24Mi            0Mi
    kube-proxy-fhvwx                          1m           39Mi            0Mi
    kube-scheduler-node01                     13m          69Mi            0Mi
    metrics-server-8598789fdb-d2kcj           5m           26Mi            0Mi

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

I recommend to review per-commit.

Does this PR introduce a user-facing change?

Added a `--show-swap` option to `kubectl top` subcommands

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md

k8s-ci-robot · 2025-01-02T16:58:20Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

sftim · 2025-01-07T13:19:59Z

Changelog suggestion

-Add a "--show-swap" option to kubectl top subcommands
+Added a `--show-swap` option to `kubectl top` subcommands

staging/src/k8s.io/kubectl/pkg/cmd/top/top_node.go

aojea · 2025-01-17T05:54:57Z

LGTM

iholder101 · 2025-03-30T12:49:46Z

If the swap feature gate is off, we should probably not display swap column.

@kannon92 I tend to disagree. I don't see a reason to disallow using the explicit --show-swap flag if the feature gate is turned off. I even think that it's beneficial, especially after turning the FG on and off (or switching between swap behaviors), to validate that indeed workloads aren't using swap. It can also be used to inspect node's swap capacity even if it is not accessible to k8s workloads.

iholder101 · 2025-03-31T14:53:26Z

currently you're always reporting 0 no matter if swap data is available or not, which might be misleading.

@soltysh Done. It will now look like the following:
With swap:

NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)   
node1   1m           10%      2Mi             10%         1Mi           0%        
node2   5m           10%      6Mi             10%         2Mi           0%        
node3   3m           10%      4Mi             10%         0Mi           0%

Withous swap:

NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)       
node1   1m           10%      2Mi             10%         unavailable   unavailable   
node2   5m           10%      6Mi             10%         unavailable   unavailable   
node3   3m           10%      4Mi             10%         unavailable   unavailable

I prefer to leave the top pod as-is TBH. It could be a useful way of ensuring pods aren't using swap after switching swap on/off, as the node will set swap capacity to nil if the feature gate is dropped:

kubernetes/pkg/kubelet/nodestatus/setters.go

Lines 359 to 363 in beef784

    
           if utilfeature.DefaultFeatureGate.Enabled(features.NodeSwap) && info.SwapCapacity != 0 { 
        
           	node.Status.NodeInfo.Swap = &v1.NodeSwapStatus{ 
        
           		Capacity: ptr.To(int64(info.SwapCapacity)), 
        
           	} 
        
           }

In addition, currently the pod printer relies solely on metrics, which only say that the pod is not using any swap. To gather information about nodes we'd need to add an addition list node API call.

iholder101 · 2025-04-21T16:06:01Z

currently you're always reporting 0 no matter if swap data is available or not, which might be misleading.

@soltysh Done. It will now look like the following: With swap:
NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)   
node1   1m           10%      2Mi             10%         1Mi           0%        
node2   5m           10%      6Mi             10%         2Mi           0%        
node3   3m           10%      4Mi             10%         0Mi           0%        
Withous swap:
NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   SWAP(bytes)   SWAP(%)       
node1   1m           10%      2Mi             10%         unavailable   unavailable   
node2   5m           10%      6Mi             10%         unavailable   unavailable   
node3   3m           10%      4Mi             10%         unavailable   unavailable   
I prefer to leave the top pod as-is TBH. It could be a useful way of ensuring pods aren't using swap after switching swap on/off, as the node will set swap capacity to nil if the feature gate is dropped:

kubernetes/pkg/kubelet/nodestatus/setters.go

Lines 359 to 363 in beef784

if utilfeature.DefaultFeatureGate.Enabled(features.NodeSwap) && info.SwapCapacity != 0 {

node.Status.NodeInfo.Swap = &v1.NodeSwapStatus{

Capacity: ptr.To(int64(info.SwapCapacity)),

}

}

In addition, currently the pod printer relies solely on metrics, which only say that the pod is not using any swap. To gather information about nodes we'd need to add an addition list node API call.

@soltysh WDYT? :)

soltysh

Two final comments, and lets land this. @iholder101 ping me on slack when you get that updated and I'll tag.

soltysh · 2025-05-26T18:04:56Z

staging/src/k8s.io/kubectl/pkg/cmd/top/top_node.go

@@ -198,6 +200,13 @@ func (o TopNodeOptions) RunTopNode() error {
 		} else {
 			availableResources[n.Name] = n.Status.Capacity
 		}
+
+		swapCapacity := int64(0)
+		if n.Status.NodeInfo.Swap != nil && n.Status.NodeInfo.Swap.Capacity != nil {


Itamar is correct here.

soltysh · 2025-05-26T18:07:21Z

staging/src/k8s.io/kubectl/pkg/metricsutil/metrics_printer.go

 		fmt.Fprintf(out, "%vMi", quantity.Value()/(1024*1024))
 	default:
 		fmt.Fprintf(out, "%v", quantity.Value())
 	}
 }

-func printPodResourcesSum(out io.Writer, total v1.ResourceList, columnWidth int) {
+func printSingleMissingResource(out io.Writer) {
+	const unavailableStr = "unavailable"


Suggested change

const unavailableStr = "unavailable"

const unavailableStr = "<unkown>"

is what we're mostly using, in kubectl decribe output, for example. So let's be consistent with that.

Signed-off-by: Itamar Holder <[email protected]>

Example output: > kubectl top node --show-swap NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) SWAP(bytes) SWAP(%) node01 500m 8% 2836Mi 60% 0Mi 0% node02 260m 5% 2206Mi 47% 512Mi 50% Signed-off-by: Itamar Holder <[email protected]>

Example output: > kubectl top pod -n kube-system --show-swap NAME CPU(cores) MEMORY(bytes) SWAP(bytes) coredns-58d5bc5cdb-5nbk4 2m 19Mi 0Mi coredns-58d5bc5cdb-jsh26 3m 37Mi 0Mi etcd-node01 51m 143Mi 0Mi kube-apiserver-node01 98m 824Mi 0Mi kube-controller-manager-node01 20m 135Mi 0Mi kube-proxy-ffgs2 1m 24Mi 0Mi kube-proxy-fhvwx 1m 39Mi 0Mi kube-scheduler-node01 13m 69Mi 0Mi metrics-server-8598789fdb-d2kcj 5m 26Mi 0Mi Signed-off-by: Itamar Holder <[email protected]>

Signed-off-by: Itamar Holder <[email protected]>

iholder101 · 2025-06-03T08:48:12Z

Thanks @soltysh!
I've addressed your feedback and rebased. PTAL :)

Signed-off-by: Itamar Holder <[email protected]>

iholder101 · 2025-06-03T14:15:26Z

/test pull-kubernetes-unit

soltysh

/lgtm
/approve

k8s-ci-robot · 2025-06-04T13:05:33Z

LGTM label has been added.

Git tree hash: 3b79e3a41287c21dd673ee47fc3ffd7d72a2f40a

k8s-ci-robot · 2025-06-04T13:05:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: iholder101, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/kubectl/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* top, refactor: turn package-exposed variables to unexpose struct fields Signed-off-by: Itamar Holder <[email protected]> * kubectl top node: add the --show-swap option Example output: > kubectl top node --show-swap NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) SWAP(bytes) SWAP(%) node01 500m 8% 2836Mi 60% 0Mi 0% node02 260m 5% 2206Mi 47% 512Mi 50% Signed-off-by: Itamar Holder <[email protected]> * kubectl top pod: add the --show-swap option Example output: > kubectl top pod -n kube-system --show-swap NAME CPU(cores) MEMORY(bytes) SWAP(bytes) coredns-58d5bc5cdb-5nbk4 2m 19Mi 0Mi coredns-58d5bc5cdb-jsh26 3m 37Mi 0Mi etcd-node01 51m 143Mi 0Mi kube-apiserver-node01 98m 824Mi 0Mi kube-controller-manager-node01 20m 135Mi 0Mi kube-proxy-ffgs2 1m 24Mi 0Mi kube-proxy-fhvwx 1m 39Mi 0Mi kube-scheduler-node01 13m 69Mi 0Mi metrics-server-8598789fdb-d2kcj 5m 26Mi 0Mi Signed-off-by: Itamar Holder <[email protected]> * kubectl top node --show-swap: add unit tests Signed-off-by: Itamar Holder <[email protected]> * kubectl top pod --show-swap: Add unit tests Signed-off-by: Itamar Holder <[email protected]> * Explicitly mark swap as unavailable when necessary Signed-off-by: Itamar Holder <[email protected]> --------- Signed-off-by: Itamar Holder <[email protected]>

k8s-ci-robot requested review from ardaguclu and mpuckett159 January 2, 2025 16:58

k8s-ci-robot added area/kubectl sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 2, 2025

iholder101 changed the title ~~kubectl top: add a --show-swap option~~ [KEP-2400] kubectl top: add a --show-swap option Jan 2, 2025

iholder101 marked this pull request as ready for review January 2, 2025 17:25

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 2, 2025

k8s-ci-robot requested review from brianpursley and soltysh January 2, 2025 17:25

sftim reviewed Jan 7, 2025

View reviewed changes

staging/src/k8s.io/kubectl/pkg/cmd/top/top_node.go Outdated Show resolved Hide resolved

iholder101 force-pushed the feature/top-swap branch from c3b0c09 to 4d75293 Compare January 7, 2025 13:23

iholder101 mentioned this pull request Jan 15, 2025

[KEP-2400] swap updates, beta3 graduation and GA criterias kubernetes/enhancements#4701

Merged

iholder101 mentioned this pull request Feb 4, 2025

[KEP-2400] Report swap capacity as part of node.status.nodeSystemInfo #129954

Merged

pacoxu mentioned this pull request Feb 24, 2025

Node memory swap support kubernetes/enhancements#2400

Open

69 tasks

iholder101 force-pushed the feature/top-swap branch 4 times, most recently from 183b011 to 63c1cd8 Compare March 18, 2025 12:54

iholder101 force-pushed the feature/top-swap branch from 2aa7eff to 3dc7e71 Compare March 30, 2025 11:34

iholder101 force-pushed the feature/top-swap branch 2 times, most recently from 51d6881 to 88b66ab Compare March 31, 2025 15:15

soltysh reviewed May 26, 2025

View reviewed changes

iholder101 force-pushed the feature/top-swap branch from 88b66ab to c9efa28 Compare June 3, 2025 08:46

iholder101 added 4 commits June 3, 2025 11:46

top, refactor: turn package-exposed variables to unexpose struct fields

e8aaeba

Signed-off-by: Itamar Holder <[email protected]>

kubectl top node: add the --show-swap option

afbdff7

Example output: > kubectl top node --show-swap NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) SWAP(bytes) SWAP(%) node01 500m 8% 2836Mi 60% 0Mi 0% node02 260m 5% 2206Mi 47% 512Mi 50% Signed-off-by: Itamar Holder <[email protected]>

kubectl top node --show-swap: add unit tests

acc25bb

Signed-off-by: Itamar Holder <[email protected]>

iholder101 force-pushed the feature/top-swap branch from c9efa28 to ed9c816 Compare June 3, 2025 08:46

kubectl top pod --show-swap: Add unit tests

8f69fde

Signed-off-by: Itamar Holder <[email protected]>

iholder101 force-pushed the feature/top-swap branch from ed9c816 to a6c8736 Compare June 3, 2025 09:17

Explicitly mark swap as unavailable when necessary

710b162

Signed-off-by: Itamar Holder <[email protected]>

iholder101 force-pushed the feature/top-swap branch from a6c8736 to 710b162 Compare June 3, 2025 13:53

soltysh approved these changes Jun 4, 2025

View reviewed changes

k8s-ci-robot assigned soltysh Jun 4, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 4, 2025

k8s-ci-robot merged commit 8d3fb9e into kubernetes:master Jun 4, 2025
14 checks passed

k8s-ci-robot added this to the v1.34 milestone Jun 4, 2025

github-project-automation bot moved this from In Progress to Done in SIG CLI Jun 4, 2025

iholder101 mentioned this pull request Jun 17, 2025

[KEP-2400] Add a container_swap_limit_bytes metric #132348

Open

	const unavailableStr = "unavailable"
	const unavailableStr = "<unkown>"

[KEP-2400] kubectl top: add a --show-swap option #129458

[KEP-2400] kubectl top: add a --show-swap option #129458

Uh oh!

Conversation

iholder101 commented Jan 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Background

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Jan 2, 2025

Uh oh!

sftim commented Jan 7, 2025

Uh oh!

Uh oh!

aojea commented Jan 17, 2025

Uh oh!

iholder101 commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iholder101 commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iholder101 commented Apr 21, 2025

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

soltysh May 26, 2025

Choose a reason for hiding this comment

Uh oh!

soltysh May 26, 2025

Choose a reason for hiding this comment

Uh oh!

iholder101 Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

iholder101 commented Jun 3, 2025

Uh oh!

iholder101 commented Jun 3, 2025

Uh oh!

soltysh left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 4, 2025

Uh oh!

k8s-ci-robot commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

iholder101 commented Jan 2, 2025 •

edited

Loading

iholder101 commented Mar 30, 2025 •

edited

Loading

iholder101 commented Mar 31, 2025 •

edited

Loading