What type of PR is this?

/kind feature
/sig node

What this PR does / why we need it:

Adds full support for node swap running on cgroup v2, for both Limited and Unlimited swap behaviors as per KEP 2400 [1].

In addition, as the KEP dictates, support for cgroup v1 is being removed.

In addition, as per the KEP, when LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled.

The formula for the swap limit for Burstable QoS pods is:
(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>.

In addition, containers with memory limits that are equal memory requests would not be able to access swap as well.
This is a nice way to opt-out of swap usage when LimitedSwap is enabled.

[1] https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md

Which issue(s) this PR fixes:

Fixes #119427
Fixes #119428
Fixes #119430

Special notes for your reviewer:

This PR introduces a commit to fix a cadvisor bug. This bug was fixed upstream, although cadvisor did not seem to have a new release since then. For testing purposes I keep the commit as being part of the PR, but it should eventually be removed in favor of bumping cadvisor's version.

UPDATE: new cadvisor version with the fix is now consumed with this PR: #119225.

Does this PR introduce a user-facing change?

Add full cgroup v2 swap support for both Limited and Unlimited swap.

When LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled.

Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.

The formula for the swap limit for Burstable QoS pods is:
`(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>`.

Support for cgroup v1 is removed.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/

Manual testing and sample output

In order to ease the review process and give an clearer idea about how this feature functions,
I'll present some sample outputs and show swap usage in action.

In the following test cases, I'll be using the following test pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: c1
    resources:
      requests:
        memory: "512M"
    image: quay.io/mabekitzur/fedora-with-stress-ng:12-3-2022
    command: ["/bin/bash"]
    args: ["-c", "sleep 9999"]

The container image is based on the latest fedora, but also installs stress-ng. This would be useful to test swap behavior.
In addition, I'm using k instead of ./cluster/kubectl.sh for simplicity.

Test case #1 - sanity check: without swap enabled

Create the pod:

> k create -f pod.yaml
pod/test-pod created

Check the swap memory limit:

> k exec -it test-pod -- "cat" "/sys/fs/cgroup/memory.swap.max"
0

See that swap is disabled (equals to 0).
Before this PR, it used to be max, which means there is no limitation for swap usage. This is fixed in this PR.

Now, let's try to stress the pod in order to show swap is being used. In order to achieve that, we can run the following two in parallel.
First, continuously print the swap usage on that Pod:

> k exec -it test-pod -- bash
[root@test-pod /]> while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 7; done

Meanwhile, let's stress the pod to force swapping out pages:

> k exec -it test-pod -- bash
[root@test-pod /]> stress-ng --vm-bytes 525158751436b --vm-keep -m 1

The current swap usage is as follows:

> k exec -it test-pod -c c1 -- bash
[root@test-pod /]# while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 2; done
0
0
...

As can be seen, swap is not being used as expected.

Case #2 - With unlimited swap

First, before deploying the cluster, the following environment variable has to be defined:

> export FEATURE_GATES="NodeSwap=true"

Create the pod:

> k create -f pod.yaml
pod/test-pod created

Check the swap memory limit:

> k exec -it test-pod -- "cat" "/sys/fs/cgroup/memory.swap.max"
max

This is now set to max as expected.

Now, let's try to stress the pod in order to show swap is being used. In order to achieve that, we can run the following two in parallel.
First, continuously print the swap usage on that Pod:

> k exec -it test-pod -- bash
[root@test-pod /]> while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 7; done

Meanwhile, let's stress the pod to force swapping out pages:

> k exec -it test-pod -- bash
[root@test-pod /]> stress-ng --vm-bytes 525158751436b --vm-keep -m 1

When we look back at the previous command continuously printing the swap usage we see:

> k exec -it test-pod -- bash
[root@test-pod /]> while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 7; done
0
# ... many zeros
0
...
2653233152
6713556992
9481170944
12596342784
16045711360
19471413248
22949982208
26457309184
30129651712
33872568320
37397729280
41005793280

As can be seen, the swap usage is getting larger and larger as time goes by.

Case #3 - With limited swap and a Burstable QoS Pod

First, before deploying the cluster, the following environment variable has to be defined:

> export FEATURE_GATES="NodeSwap=true"
> export LIMITED_SWAP=true

Note that LIMITED_SWAP is a new environment variable introduced in this PR.

Create the pod:

> k create -f pod.yaml
pod/test-pod created

As per KEP2400, the swap limit is being automatically calculated for Burstable QoS pods. As written in the KEP, the fomula is: (<container-request>/<node-memory-capacity>)*<swap-size>). On my system, I have approximately 400Gi of total memory and a swap size of approximately 40Gi. Therefore, the swap limit needs to be approximately (0.5Gi/400Gi)*40Gi == 0.5Gi.

Let's check the memory limit that was set automatically:
Check the swap memory limit:

> k exec -it test-pod -- "cat" "/sys/fs/cgroup/memory.swap.max"
54435840

Since 54,435,840 bytes == (54,435,840/1024^3)Gi ~= 0.5Gi, we can tell that the limit was set as expected.

Now, as before, let's stress the pod and see the actual swap usage:

> k exec -it test-pod -- bash
[root@test-pod /]> while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 7; done
0
# ... many zeros
0
...
54435840
54435840
54435840
54435840
...

As we can see, the pod cannot exceed its swap limit, as expected.

Case #4 - With limited swap and a Burstable QoS Pod with memory limits==requests

In this case the KEP dictates that no swap memory should be allocated to the container.
This way it's easy to opt-out of swap.

Note: the same behavior would apply to Best-Effort / Guaranteed QoS Pods.
Note: for this example I've changed the pod example above so that memory limits are equal to requests

As before, we'll depoloy the cluster with NodeSwap feature gate and LimitedSwap and create the pod:

> export FEATURE_GATES="NodeSwap=true"
> export LIMITED_SWAP=true

> k create -f pod.yaml
pod/test-pod created

Check the swap memory limit:

> k exec -it test-pod -- "cat" "/sys/fs/cgroup/memory.swap.max"
0

See that swap is disabled (equals to 0).

If we would repeat the same steps as before, meaning stressing the container:

> k exec -it test-pod -c c1 -- bash
[root@test-pod /]# while true; do cat /sys/fs/cgroup/memory.swap.current; sleep 2; done
0
0
...

As can be seen, swap is not being used as expected.