Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy: fix combination of --config and logging command line flags #119867

Merged
merged 1 commit into from
Aug 10, 2023

Conversation

pohly
Copy link
Contributor

@pohly pohly commented Aug 9, 2023

What type of PR is this?

/kind bug
/kind regression

What this PR does / why we need it:

When parsing a config file, all settings derived from command line flags are discarded because only the config settings are used. That has been the traditional behavior for non-logging flags.

But --config ... -v=4 used to work until
71ef0da added logging to the configuration. To restore the original behavior, kube-proxy now:

  • parses flags
  • reads the config file
  • applies logging settings from the flags to the config loaded from file
  • uses that merged config

Which issue(s) this PR fixes:

Fixes #119864

Special notes for your reviewer:

Needs more tests...

Does this PR introduce a user-facing change?

kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the `-v` command line flag when combined with `--config`.

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. labels Aug 9, 2023
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Wed Aug 9 10:32:39 UTC 2023.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 9, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/kube-proxy kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 9, 2023
// SetFlags applies the logging flags from the given flag set to the given
// configuration. Fields for which the corresponding flag was not used are left
// unmodified.
func SetFlags(fs *pflag.FlagSet, c *LoggingConfiguration) error {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not happy with the name.

Perhaps CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want to make this part of the log api?

should not handle this directly in kube-proxy since it already has a precedent overriding a flag #119864 (comment)

This way you can come with a more universal solution in 1.28

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the final solution may be something like this. Doing this entirely in kube-proxy as a hot fix now is easier than trying to get it implemented properly here.

OTOH, this is just a Go API. It can be changed at any time without breaking the API of components using it, only compilation of the component may break...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

your call, I just wanted to give you more options, with kube-proxy I think that is fair to go with the override in 1.28

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)?

OverrideLogsFromFlags(from *pflag.FlagSet, to *LoggingConfiguration) ?

// SetFlags applies the logging flags from the given flag set to the given
// configuration. Fields for which the corresponding flag was not used are left
// unmodified.
func SetFlags(fs *pflag.FlagSet, c *LoggingConfiguration) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)?

OverrideLogsFromFlags(from *pflag.FlagSet, to *LoggingConfiguration) ?

var cloneFS pflag.FlagSet
addFlags(c, &cloneFS)
var err error
cloneFS.VisitAll(func (f *pflag.Flag) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cloneFS.VisitAll(func (f *pflag.Flag) {
cloneFS.VisitAll(func(f *pflag.Flag) {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For --vmodule, the flag doesn't override: instead, the entries from the config and the command line get merged, with command line entries at the beginning where they have priority. I think that's the right semantic.

But therefore "override" isn't quite appropriate, so I'm going with "copy".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, got it.

@cyclinder
Copy link
Contributor

/test pull-kubernetes-e2e-kind

@cyclinder
Copy link
Contributor

I did some tests based on your changes, It works well. Maybe we need some comments from aojea, as mentioned by #119864 (comment).

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 10, 2023
@furkatgofurov7
Copy link
Member

furkatgofurov7 commented Aug 10, 2023

@aojea @cyclinder is this PR should be labelled with v1.28 milestone?

When parsing a config file, all settings derived from command line flags are
discarded because only the config settings are used. That has been the
traditional behavior for non-logging flags.

But `--config ... -v=4` used to work until
71ef0da added logging to the configuration.
To restore the original behavior, kube-proxy now:
- parses flags
- reads the config file
- applies logging settings from the flags to the config loaded from file
- uses that merged config
// unmodified. For fields that have multiple values (like vmodule), the values from
// the flags get joined so that the command line flags have priority.
//
// TODO (pohly): move this to logsapi
Copy link
Contributor

@cyclinder cyclinder Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this after the 1.28 release?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it has to be later

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's to it in 1.29.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it be moved now? I can work on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and please do.

Some points to consider:

  • Can the same be done for apps using just plain Go flags? The "flag was set" field only exists for pflag. A potential solution might be to wrap the values in a flag.Value implementation which tracks whether Set was called.
  • Examples and docs...
    • The original plan was to have a JSON or YAML config that gets decoded into the LoggingConfiguration instance after parsing, in which case this new function isn't needed. It needs to be checked that this works and be documented.
    • For configs where that doesn't work (i.e. anything based on registered types via apimachinery like this config here), the new functions need to be documented.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A potential solution might be to wrap the values in a flag.Value implementation which tracks whether Set was called.

I found it hard to check if a go flag was set, I tried to check it by using the following codes, but it didn't work.

func isFlagSet(name string) bool {
    found := false
    flag.Visit(func(f *flag.Flag) {
        if f.Name == name {
            found = true
        }
    })
    return found
}

Could you give me an example?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and we may need a new function called copyLogsFromGoFlags for this, It may not be for Kubernetes users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A wrapper as in this example is one possible solution:

package main

import (
	"flag"
	"fmt"
)

type flagWrapper struct {
	flag.Value
	isSet bool
}

func (f *flagWrapper) Set(value string) error {
	f.isSet = true
	return f.Value.Set(value)
}

var _ flag.Value = &flagWrapper{}

func main() {
	var fs flag.FlagSet
	fs.Int("some-int", 42, "the answer")
	arg := fs.Lookup("some-int")
	wrapper := &flagWrapper{Value: arg.Value}
	flag.CommandLine.Var(wrapper, arg.Name, arg.Usage)
	flag.Parse()
	fmt.Printf("some-int set = %v\n", wrapper.isSet)
}

The downside is that -h doesn't know about the actual type of the parameter:

$ go run . -h
Usage of /tmp/go-build544671316/b001/exe/testflag:
  -some-int value
    	the answer

Without the wrapper, int would be shown instead of value. That's because flag looks at the type of a value and has special support for builtin value types.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the example.

The downside is that -h doesn't know about the actual type of the parameter

Can we explain in the usage what the actual type of the parameter is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Showing "value" isn't a showstopper.

@aojea
Copy link
Member

aojea commented Aug 10, 2023

/milestone v1.28

@k8s-ci-robot k8s-ci-robot added this to the v1.28 milestone Aug 10, 2023
@cyclinder
Copy link
Contributor

@furkatgofurov7 I think yes, but need ack for @pohly @aojea

// command line flags have priority). Otherwise `--config
// ... -v=5` doesn't work (config resets verbosity even
// when it contains no logging settings).
copyLogsFromFlags(fs, &c.Logging)
Copy link
Member

@aojea aojea Aug 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this scopes the change to the Logging configuration that makes this change contained

+1

@@ -235,7 +235,7 @@ func NewOptions() *Options {
}

// Complete completes all the required options.
func (o *Options) Complete() error {
func (o *Options) Complete(fs *pflag.FlagSet) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we make the Flags part of the Options struct instead of plumbing it down?
I see kube-scheduler has them there too, it feels like is cleaner

// Flags hold the parsed CLI flags.
Flags *cliflag.NamedFlagSets

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but that would be a larger change that I don't want to do this close to a release.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

Comment on lines +282 to +304
logsapi.AddFlags(to, &cloneFS)
vmodule := to.VModule
to.VModule = nil
var err error
cloneFS.VisitAll(func(f *pflag.Flag) {
if err != nil {
return
}
fsFlag := from.Lookup(f.Name)
if fsFlag == nil {
err = fmt.Errorf("logging flag %s not found in flag set", f.Name)
return
}
if !fsFlag.Changed {
return
}
if setErr := f.Value.Set(fsFlag.Value.String()); setErr != nil {
err = fmt.Errorf("copying flag %s value: %v", f.Name, setErr)
return
}
})
to.VModule = append(to.VModule, vmodule...)
return err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't really evaluate this logic, I trust other reviewers

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand it correctly it overrides everything in logging present in flags, except the to.Vmodule that gets appended

@aojea
Copy link
Member

aojea commented Aug 10, 2023

/approve

one question about the code structure #119867 (comment), if we make flags part of the options struct I think it will make easier future changes , in case we no longer need to pass the flags to the Complete method per example

I trust @cyclinder or @liggitt and the unit test for this #119867 (comment)

@pohly can you remove the WIP from the title if this is ready

/assign @liggitt

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2023
@pohly pohly changed the title WIP: kube-proxy: fix combination of --config and logging command line flags kube-proxy: fix combination of --config and logging command line flags Aug 10, 2023
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 10, 2023
@furkatgofurov7
Copy link
Member

Also tagging relevant SIG based on the label on the PR

@kubernetes/sig-network-leads

@aojea
Copy link
Member

aojea commented Aug 10, 2023

Also tagging relevant SIG based on the label on the PR

@kubernetes/sig-network-leads

I'm approving for sig-network

kubernetes/OWNERS_ALIASES

Lines 255 to 259 in a287fd6

sig-network-approvers:
- andrewsykim
- aojea
- bowei
- caseydavenport

@aojea
Copy link
Member

aojea commented Aug 10, 2023

/lgtm

I think this is ok, the change is very contained and well covered with unit tests, and we can see also in CI is working fine

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 10, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: c0b3740e5310a03e449b9dd9bd5dbc0dd45b0794

@k8s-ci-robot k8s-ci-robot merged commit 18c3c7e into kubernetes:master Aug 10, 2023
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kube-proxy cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

kube-proxy: -v command line flag reset by --config file
6 participants