Skip to content

Add emulation version info to client-go's user-agent #131401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Add emulation version info to client-go's user-agent #131401

wants to merge 1 commit into from

Conversation

richabanker
Copy link
Contributor

@richabanker richabanker commented Apr 21, 2025 •
edited
Loading

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds Emulation Version and Min-Compat Version info to client-go's UserAgent (currently only has Binary Version) which is propagated as a request header

Debug logs: 


Scheduler:
request header: map[Accept:[application/vnd.kubernetes.protobuf, */*] Content-Length:[443] Content-Type:[application/vnd.kubernetes.protobuf] User-Agent:[kube-scheduler/{BinaryVersion: 1.34.0, EmulationVersion: 1.34, MinCompatibilityVersion: 1.33}/ (linux/amd64) kubernetes/666c634/leader-election]]

Kube-controller-manager: 
request header: map[Accept:[application/vnd.kubernetes.protobuf, */*] Content-Length:[461] Content-Type:[application/vnd.kubernetes.protobuf] User-Agent:[kube-controller-manager/{BinaryVersion: 1.34.0, EmulationVersion: 1.34, MinCompatibilityVersion: 1.33}/ (linux/amd64) kubernetes/666c634/leader-election]]

Kube-probe: 
request header: map[Accept:[*/*] User-Agent:[kube-probe/1.34+]]

Kubelet: 
request header: map[Accept:[application/vnd.kubernetes.protobuf,application/json] Accept-Encoding:[gzip] Content-Length:[534] Content-Type:[application/vnd.kubernetes.protobuf] User-Agent:[kubelet/v1.34.0 (linux/amd64) kubernetes/666c634]]

Which issue(s) this PR fixes:

This is needed to have a request level metric which tracks number of out-of-skew requests received by the apiserver. This is one of the beta requirements for https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/4330-compatibility-versions , ref

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot
Copy link
Contributor

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 21, 2025
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.33 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.33.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Mon Apr 21 13:37:44 UTC 2025.

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 21, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/apiserver labels Apr 21, 2025
@k8s-ci-robot k8s-ci-robot requested review from dom4ha and thockin April 21, 2025 21:24
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 21, 2025
@richabanker richabanker force-pushed the compat-version-skew branch from 238bc3d to d99cd2d Compare April 23, 2025 16:56
@richabanker
Copy link
Contributor Author

@siyuanfoundation

dom4ha
dom4ha reviewed May 14, 2025
View reviewed changes
Copy link
Member

@dom4ha dom4ha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide more details (in the description) why the change is needed and why to the scheduler only.

cmd/kube-scheduler/app/options/options.go Outdated
client, err := clientset.NewForConfig(restclient.AddUserAgent(kubeConfig, "scheduler"))
func createClients(kubeConfig *restclient.Config, componentRegistry basecompatibility.ComponentGlobalsRegistry) (clientset.Interface, clientset.Interface, error) {
effectiveVer := componentRegistry.EffectiveVersionFor(basecompatibility.DefaultKubeComponent)
compatVersionInfo := fmt.Sprintf("%s=%s", "emulationVersion", effectiveVer.EmulationVersion())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be empty (not reported) if there is no emulation version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised it, PTAL now.

@richabanker richabanker force-pushed the compat-version-skew branch 5 times, most recently from 48c8a7d to 40b668d Compare June 3, 2025 16:44
@jpbetz
Copy link
Contributor

jpbetz commented Jun 5, 2025

@Jefftree would you review for emulation version bits?

@richabanker Can we find some way to harness this with a test?

@richabanker richabanker force-pushed the compat-version-skew branch 3 times, most recently from a615213 to cea44bc Compare June 25, 2025 05:15
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: richabanker
Once this PR has been reviewed and has the lgtm label, please assign dom4ha, sttts for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@richabanker richabanker force-pushed the compat-version-skew branch from cea44bc to 6771a0e Compare June 25, 2025 05:18
@richabanker richabanker force-pushed the compat-version-skew branch from 6771a0e to 38ecdc5 Compare June 25, 2025 05:19
@richabanker
Copy link
Contributor Author

richabanker commented Jun 25, 2025 •
edited
Loading

The version info added here in the request header will eventually be used to record an apiserver metric that tells the number of out-of-skew requests seen so far, a concern raised by @liggitt here was that

"user-agent headers are totally under the client's control... and don't drive any behavior... feels a little weird to base apiserver metrics on them

curl ... -H "User-Agent: ... kube-controller-manager/1.16.0 emulate=1.16.0 could just break users of the metric"

Do we want to re-evaluate whether clients sending version info in request headers is the right way to go about this?

@deads2k for your thoughts since this requirement came from this discussion on the KEP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dom4ha dom4ha dom4ha left review comments

@thockin thockin Awaiting requested review from thockin

Assignees
No one assigned
Labels
area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@richabanker @k8s-ci-robot @jpbetz @dom4ha