Add metric for e2e pod startup latency including image pull #121041
Conversation
|
@ruiwen-zhao: GitHub didn't allow me to request PR reviews from the following users: elfinhe. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
pkg/kubelet/metrics/metrics.go
Outdated
| @@ -162,7 +163,7 @@ var ( | |||
| Subsystem: KubeletSubsystem, | |||
| Name: PodStartDurationKey, | |||
| Help: "Duration in seconds from kubelet seeing a pod for the first time to the pod starting to run", | |||
| Buckets: metrics.DefBuckets, | |||
| Buckets: []float64{0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600}, | |||
There was a problem hiding this comment.
yeah, the buckets here are the same as pod_start_sli_duration_seconds. We need the super long buckets because there might be node creation (in case of GKE's autoscaling, for example), stockout, retries, between a pod being created and a pod starting. All of those could take a long time.
|
/triage accepted |
|
/retest |
724789d
to
832c1b2
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mrunalp, ruiwen-zhao The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
a44db93
to
7bed864
Compare
pkg/kubelet/metrics/metrics.go
Outdated
| @@ -41,6 +41,7 @@ const ( | |||
| PodWorkerDurationKey = "pod_worker_duration_seconds" | |||
| PodStartDurationKey = "pod_start_duration_seconds" | |||
| PodStartSLIDurationKey = "pod_start_sli_duration_seconds" | |||
| PodStartE2EDurationKey = "pod_start_e2e_duration_seconds" | |||
There was a problem hiding this comment.
nit for naming: how about pod_start_total_duration_seconds?
There was a problem hiding this comment.
I perfer e2e mostly because it sounds more "e2e", and it aligns with the name of the slo-monitor metric: https://github.com/kubernetes/perf-tests/blob/358844d5ba650dd5a57bae0072691d93cd82e1bd/slo-monitor/src/monitors/pod_monitor.go#L43
There was a problem hiding this comment.
e2e in kubernetes has a hard bias towards e2e tests, I suggest other name less confusing
There was a problem hiding this comment.
makes sense. I changed it to pod_start_total_duration_seconds as Qiutong suggests.
|
/lgtm |
|
LGTM label has been added. Git tree hash: a465d8c3e39844efb2c251eab9524b9da6c9636c
|
Signed-off-by: ruiwen-zhao <[email protected]>
7bed864
to
1165609
Compare
|
/lgtm Awesome work, we have eyes now :) |
|
LGTM label has been added. Git tree hash: fb9e39b996f9985771727b6b4cd87f5a7f63b0ad
|
What type of PR is this?
/kind feature
What this PR does / why we need it:
Currently Pod startup latency tracker only emits a latency that excludes image pull time. This latency might not reflect what users actually see because image pull is a major part of the overall latency. Therefore, I am adding a metric for e2e latency that includes the image pull time, so that we can:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: