collect watch-list requests latency metric #120490

p0lyn0mial · 2023-09-07T12:12:01Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Collects watch-list requests latency.
Having this metric allows for comparing watch-list requests with standard list requests and measuring performance of the new requests in general.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

xref: kubernetes/enhancements#3157

Does this PR introduce a user-facing change?

Adds `apiserver_watch_list_duration_seconds` metrics. Which will measure response latency distribution in seconds for watch list requests broken by group, version, resource and scope

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

p0lyn0mial · 2023-09-07T12:12:19Z

/assign @wojtek-t @dgrisonnet

p0lyn0mial · 2023-09-07T12:15:33Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go

@@ -146,6 +146,18 @@ var (
 },
 []string{"resource"},
 )
+
+ WatchListLatencies = compbasemetrics.NewHistogramVec(


@dgrisonnet could please take a look at this metric ?

My intention here is to measure latency of a special type of watch requests called watch-list.
I'd like to compare it against standard list requests to see how the new mechanism performs.

The standard list requests are measured by

kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

Line 91 in c607984

requestLatencies = compbasemetrics.NewHistogramVec(

I'm not sure if the Buckets need to match or not.

Is there a way to differentiate watch list requests with different scopes and is it relevant here? By scope I mean either:

resource: if the request is about a single object

namespace: if it is about objects from a single namespace

cluster: if it spawns objects from multiple namespaces

Similar to what we have today when we measure the latency of requests to the kube-apiserver. The idea would be to differentiate the latency when watching a single object vs watching all of them. This would be essential in my opinion in order to have precise measurements and to be able to properly compare the latency of watch list requests to the LIST requests.

okay, thanks, sgtm, I will add the scope label to the metrics.

I opened #120520 as it will make the logic that determines the scope of a request easier.

I've also opened kubernetes/perf-tests#2315 that will allow us to measure the new metric in the perf test

dgrisonnet · 2023-09-07T15:27:53Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go

+ Subsystem: subsystem,
+ Name: "watch_list_duration_seconds",
+ Help: "Response latency distribution in seconds for watch list requests broken by group, resource.",
+ Buckets: []float64{0.5, 1.0, 2, 5, 8, 10, 15, 20, 30, 45, 60},


Bucket don't necessarily need to match to be able to compare to the kube-apiserver metrics, but isn't 0.5 second quite high for resource-scoped watch list requests?

maybe @wojtek-t can comment on that. I don't have experience with actual numbers.

Agree - maybe we don't need all the buckets from:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L99

but I would definitely start with something lower (0.025?) and just took a subset of those

wojtek-t · 2023-09-14T07:27:12Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go

@@ -313,6 +343,7 @@ func (c *cacheWatcher) markBookmarkAfterRvSent(event *watchCacheEvent) {
 defer c.stateMutex.Unlock()
 if !c.wasBookmarkAfterRvSentLocked() {
 c.state = cacheWatcherBookmarkSent
+ c.metrics.CalculateAndRecordWatchListRequestLatency()


What I'm a bit worried is that we will be comparing apples to oranges.

The current metric (apiserver_request_latencies) includes that time it takes to send the result to the client too. Here we effectively only measure time to generate them.
I'm not sure how to fix this, but it's a bit concerning to me.

What I started wondering is whether we actually want to measure the latency here.

Wouldn't it be better to actually measure the latency at the higher layers of the stack?
What if instead we would measure it somewhere here:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go#L183

?

This is a good point, I wasn't aware that there is a layer above this one where we could measure the latency.
Let me rework this PR then. Thanks!

wojtek-t · 2023-09-14T07:30:32Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher_test.go

@@ -650,3 +682,85 @@ func TestBookmarkAfterResourceVersionWatchers(t *testing.T) {
 t.Fatalf("expected only one watcher to be expired")
 }
 }
+
+func TestWatchListLatency(t *testing.T) {


This test seems somewhat artificial to me - I'm not sure I see an actual value in it...

wojtek-t · 2023-09-14T07:32:55Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go

+ Subsystem: subsystem,
+ Name: "watch_list_duration_seconds",
+ Help: "Response latency distribution in seconds for watch list requests broken by group, resource.",
+ Buckets: []float64{0.5, 1.0, 2, 5, 8, 10, 15, 20, 30, 45, 60},


Agree - maybe we don't need all the buckets from:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L99

but I would definitely start with something lower (0.025?) and just took a subset of those

wojtek-t · 2023-09-14T07:33:00Z

staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go

+ Buckets: []float64{0.5, 1.0, 2, 5, 8, 10, 15, 20, 30, 45, 60},
+ StabilityLevel: compbasemetrics.ALPHA,
+ },
+ []string{"group", "resource", "scope"},


and version?

wojtek-t · 2023-09-19T09:31:20Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -248,7 +255,14 @@ func (s *WatchServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 }
 metrics.WatchEvents.WithContext(req.Context()).WithLabelValues(kind.Group, kind.Version, kind.Kind).Inc()

- if err := s.EmbeddedEncoder.Encode(event.Object, buf); err != nil {
+ if event.Type == watch.Bookmark && utilfeature.DefaultFeatureGate.Enabled(features.WatchList) {


Checking quickly if this will somehow mess up with #120300 but I think it doesn't make it visibly harder.

Might be worth checking additionally.

yeah, it doesn't it will be an easy merge :)

wojtek-t · 2023-09-19T09:36:30Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -248,7 +255,14 @@ func (s *WatchServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 }
 metrics.WatchEvents.WithContext(req.Context()).WithLabelValues(kind.Group, kind.Version, kind.Kind).Inc()

- if err := s.EmbeddedEncoder.Encode(event.Object, buf); err != nil {
+ if event.Type == watch.Bookmark && utilfeature.DefaultFeatureGate.Enabled(features.WatchList) {
+ shouldRecordWatchListLatency, err = storage.HasInitialEventsEndBookmarkAnnotation(event.Object)


OK - so what you're doing is not even trying to understand whether user requests initial-list etc.
Instead you're assuming that lower layers will do the job and send (exactly one) initial-events-end bookmark correctly and just rely on that.

I guess I'm fine with that, but I would add a comment explaining that here.

yes, it is a safe assumption, okay, I will add a comment.

wojtek-t · 2023-09-19T09:38:05Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

+ utilruntime.HandleError(fmt.Errorf("unable to measure watchlist latency because no req info found in the ctx, gvr: %s", reqScope.Resource))
+ return
+ }
+ // TODO(p0lyn0mial): figure out how to get ns and name for watch requests


Why? We don't need name/namespace - we are only reporting it per scope...

Yes, this is required for the scope. I'm pretty sure it will work most of the time.
Except one case mentioned in #120520 (comment)

The worst case scenario would be that we end up with an empty scope

OK - I see what you're saying now.

If you could open a separate PR with tests that Jordan mentioned and a fix for the missing namespace?

yes, I was planning to add support for it in kubernetes/perf-tests#2316

wojtek-t · 2023-09-19T09:44:25Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

+ if err != nil {
+ utilruntime.HandleError(fmt.Errorf("unable to determine if the obj has the required annotation for measuring watchlist latency, obj %T: %v", event.Object, err))
+ }
+ }


nit: I think I would prefer extracting this so a simple helper function:

func shouldRecordWatchListLatency(event) bool { if event.Type != Bookmark || !utilfeature.... { return fasle } hasAnnotation, err := ... if err != nil { utilruntime.HandleError(...) return false } return true }

wojtek-t · 2023-09-19T09:46:26Z

staging/src/k8s.io/apiserver/pkg/storage/util.go

+ }
+ objAnnotations := objMeta.GetAnnotations()
+ _, ok := objAnnotations[initialEventsAnnotationKey]
+ return ok, nil


To be on the safe-side, should we actually check if the value is "true" ?

I was thinking about it as well but then I realised that we didn't bother to check it for client-go either.
Let me open a PR addressing it for client-go sot that we are on the safe-side.

ptal, #120752

p0lyn0mial · 2023-09-19T11:15:16Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+ Name: "watch_list_duration_seconds",
+ Help: "Response latency distribution in seconds for watch list requests broken by group, version, resource and scope.",
+ // TODO(p0lyn0mial): adjust the buckets
+ Buckets: []float64{0.5, 1.0, 2, 5, 8, 10, 15, 20, 30, 45, 60},


Agree - maybe we don't need all the buckets from:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L99

but I would definitely start with something lower (0.025?) and just took a subset of those

@wojtek-t I think that 0.025 is 25 milliseconds, which seems to be too low, no ? did you mean 0.25 :) ?

I meant 0.025 - for very small lists it's not unreasonable. Maybe 0.05 - but I want to see some percent of those to be below 100ms eventually (once Marek's improvements with manual progress notify land).

below 100ms eventually

wow, nice :) I can add 0.05 in addition to what we have atm.

can you explain what Marek will be working on ?

dgrisonnet · 2023-09-19T20:17:59Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -280,6 +288,10 @@ func (s *WatchServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 if len(ch) == 0 {
 flusher.Flush()
 }
+ if isWatchListLatencyRecordingRequired {
+ recordWatchListLatency(req.Context(), s.Scope)
+ isWatchListLatencyRecordingRequired = false


I might be missing something, but this doesn't seem to have any effect does it?

oh, you are right, I can remove it now. thanks!

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

p0lyn0mial · 2023-09-25T11:22:43Z

@dgrisonnet @wojtek-t this PR is ready for another pass, ptal, thanks!

wojtek-t · 2023-09-25T11:26:09Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -61,7 +68,7 @@ func (w *realTimeoutFactory) TimeoutCh() (<-chan time.Time, func() bool) {

 // serveWatch will serve a watch response.
 // TODO: the functionality in this method and in WatchServer.Serve is not cleanly decoupled.
-func serveWatch(watcher watch.Interface, scope *RequestScope, mediaTypeOptions negotiation.MediaTypeOptions, req *http.Request, w http.ResponseWriter, timeout time.Duration) {
+func serveWatch(watcher watch.Interface, scope *RequestScope, mediaTypeOptions negotiation.MediaTypeOptions, req *http.Request, w http.ResponseWriter, timeout time.Duration, opts *metainternalversion.ListOptions) {


Rather than passing ListOptions here, I would prefer to actually pass the already computed metricsScope value.

okay, ic, this would be computed by the CleanListScope function. I'm fine with that as well.

wojtek-t · 2023-09-25T11:29:22Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

+//
+// note that normally we would use RequestInfo for computation
+// bur we cannot, due to the same reasons mentioned in determineRequestNamespaceAndName
+func determineRequestScope(ctx context.Context, opts *metainternalversion.ListOptions) string {


We should move this function to pkg/endpoint/metrics/metrics.go [there is CleanScope there as an example]

Maybe let's call it CleanListScope or sth like that....

We could do that. So I assume you would also like to move the determineRequestNamespaceAndName function to the same place, right ?

moved, ptal.

wojtek-t · 2023-09-25T13:28:24Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+ opts.FieldSelector = fields.Nothing()
+ } else if opts.FieldSelector == nil {
+ opts.FieldSelector = fields.Nothing()
+ }


Let's not modify opts here - see my comments below.

wojtek-t · 2023-09-25T13:29:16Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+
+ if requestNamespace, ok := request.NamespaceFrom(ctx); ok && len(requestNamespace) > 0 {
+ namespace = requestNamespace
+ } else if selectorNamespace, ok := opts.FieldSelector.RequiresExactMatch("metadata.namespace"); ok {


else if opts != nil && opts.FieldSelector != nil { if selectorNamespace, .... }

wojtek-t · 2023-09-25T13:29:24Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+ }
+ if requestInfo, ok := request.RequestInfoFrom(ctx); ok && requestInfo != nil && len(requestInfo.Name) > 0 {
+ name = requestInfo.Name
+ } else if selectorName, ok := opts.FieldSelector.RequiresExactMatch("metadata.name"); ok {


wojtek-t · 2023-09-25T13:30:01Z

staging/src/k8s.io/apiserver/pkg/storage/util.go

@@ -29,6 +29,13 @@ import (
 "k8s.io/apimachinery/pkg/runtime"
 )

+const (


Let's move this one to a separate cleanup PR (together with the new func and test).

I will rebase this PR once 120872 merges unless git is able to figure it out automatically.

wojtek-t · 2023-09-25T13:31:53Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -360,3 +374,31 @@ func (s *WatchServer) HandleWS(ws *websocket.Conn) {
 }
 }
 }
+
+func recordWatchListLatency(ctx context.Context, reqScope *RequestScope, metricsScope string) {


Move it to pkg/endpoints/metrics/metrics.go - it's where a bunch of Record* functions are sitting.

dgrisonnet

The new metric looks good to me 👍

p0lyn0mial · 2023-09-25T15:42:11Z

@dgrisonnet @wojtek-t rebased against the master branch. This PR is ready for a final review.

p0lyn0mial · 2023-09-26T10:47:27Z

/retest

wojtek-t · 2023-09-26T15:56:35Z

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

@@ -219,6 +226,8 @@ func (s *WatchServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
 w.WriteHeader(http.StatusOK)
 flusher.Flush()

+ var err error
+ var isWatchListLatencyRecordingRequired bool


nit: let's not define them here and instead just revert to create them as we were creating them before in each loop [this doesn't bring value for simple types]

Could be that the go's compiler implements loop invariant code motion optimisation anyway.
I think that you are right that declaring a simple variable like bool on every iteration may not have a significant impact on the GC. Sure I can inline these variables.

wojtek-t · 2023-09-26T16:00:11Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+//
+// Note that normally we would use CleanScope for computation.
+// But due to the same reasons mentioned in determineRequestNamespaceAndName we cannot.
+func CleanListScope(ctx context.Context, opts *metainternalversion.ListOptions) string {


Can you please add unit tests for this function?

wojtek-t · 2023-09-26T16:04:41Z

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go

+ Subsystem: APIServerComponent,
+ Name: "watch_list_duration_seconds",
+ Help: "Response latency distribution in seconds for watch list requests broken by group, version, resource and scope.",
+ Buckets: []float64{0.1, 0.25, 0.5, 1.0, 2, 5, 10, 20, 30, 45},


I still think that we want to use more buckets, and more aligned request_latencies metric which are:

{0.005, 0.025, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3, 4, 5, 6, 8, 10, 15, 20, 30, 45, 60

We can probably drop some values, but especially given we will have progress-notify eventually, I would like to have better comparison with lists, so I suggest:

{0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 2, 4, 6, 8, 10, 15, 20, 30, 45, 60}

sure, I guess we are about to find out what values really matter, i don't have strong opinions here.

alexzielenski · 2023-09-26T20:08:44Z

/triage accepted

…nctions

wojtek-t · 2023-09-27T12:36:22Z

/lgtm
/approve

k8s-ci-robot · 2023-09-27T12:36:29Z

LGTM label has been added.

Git tree hash: e37034ddebcac27ae667fe0f7370045bd7efc88f

k8s-ci-robot · 2023-09-27T12:36:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, p0lyn0mial, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/pkg/endpoints/OWNERS~~ [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot assigned dgrisonnet and wojtek-t Sep 7, 2023

k8s-ci-robot requested review from MadhavJivrajani and mengjiao-liu September 7, 2023 12:13

p0lyn0mial commented Sep 7, 2023

View reviewed changes

dgrisonnet reviewed Sep 7, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from fdf745b to 8738bce Compare September 8, 2023 15:46

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 8, 2023

p0lyn0mial mentioned this pull request Sep 11, 2023

introduce WatchListLatencyPrometheus measurement kubernetes/perf-tests#2315

Open

wojtek-t reviewed Sep 14, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from 8738bce to 47e5ee6 Compare September 19, 2023 08:02

wojtek-t reviewed Sep 19, 2023

View reviewed changes

p0lyn0mial commented Sep 19, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from 47e5ee6 to b72df9d Compare September 19, 2023 11:47

dgrisonnet reviewed Sep 19, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from b72df9d to c108f85 Compare September 25, 2023 11:17

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Sep 25, 2023

p0lyn0mial changed the title ~~WIP: collect watch-list requests latency metric~~ collect watch-list requests latency metric Sep 25, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 25, 2023

wojtek-t reviewed Sep 25, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from c108f85 to 0fb7168 Compare September 25, 2023 12:32

wojtek-t reviewed Sep 25, 2023

View reviewed changes

p0lyn0mial mentioned this pull request Sep 25, 2023

storage/util: introduce HasInitialEventsEndBookmarkAnnotation #120872

Merged

dgrisonnet approved these changes Sep 25, 2023

View reviewed changes

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch 2 times, most recently from 21dde8f to 2e2f9ce Compare September 25, 2023 15:41

wojtek-t reviewed Sep 26, 2023

View reviewed changes

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 26, 2023

p0lyn0mial added 2 commits September 27, 2023 09:03

handlers/watch: calculate and record WatchList latency metric.

772b1f4

endpoints/metrics: define watchListLatencies metric and associated fu…

a97f4b7

…nctions

p0lyn0mial force-pushed the upstream-watch-list-cache-metrics branch from 2e2f9ce to a97f4b7 Compare September 27, 2023 07:30

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 27, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 27, 2023

k8s-ci-robot merged commit bdedc21 into kubernetes:master Sep 27, 2023
14 checks passed

k8s-ci-robot added this to the v1.29 milestone Sep 27, 2023

collect watch-list requests latency metric #120490

collect watch-list requests latency metric #120490

Conversation

p0lyn0mial commented Sep 7, 2023 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

p0lyn0mial commented Sep 7, 2023

p0lyn0mial Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgrisonnet Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0lyn0mial commented Sep 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgrisonnet left a comment

Choose a reason for hiding this comment

p0lyn0mial commented Sep 25, 2023

p0lyn0mial commented Sep 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexzielenski commented Sep 26, 2023

wojtek-t commented Sep 27, 2023

k8s-ci-robot commented Sep 27, 2023

k8s-ci-robot commented Sep 27, 2023

p0lyn0mial commented Sep 7, 2023 •

edited

p0lyn0mial Sep 7, 2023 •

edited

dgrisonnet Sep 7, 2023 •

edited