-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kube-proxy: fix combination of --config and logging command line flags #119867
Conversation
Please note that we're already in Test Freeze for the Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Wed Aug 9 10:32:39 UTC 2023. |
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
// SetFlags applies the logging flags from the given flag set to the given | ||
// configuration. Fields for which the corresponding flag was not used are left | ||
// unmodified. | ||
func SetFlags(fs *pflag.FlagSet, c *LoggingConfiguration) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not happy with the name.
Perhaps CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you want to make this part of the log api?
should not handle this directly in kube-proxy since it already has a precedent overriding a flag #119864 (comment)
This way you can come with a more universal solution in 1.28
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the final solution may be something like this. Doing this entirely in kube-proxy as a hot fix now is easier than trying to get it implemented properly here.
OTOH, this is just a Go API. It can be changed at any time without breaking the API of components using it, only compilation of the component may break...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
your call, I just wanted to give you more options, with kube-proxy I think that is fair to go with the override in 1.28
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps
CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)
?
OverrideLogsFromFlags(from *pflag.FlagSet, to *LoggingConfiguration) ?
// SetFlags applies the logging flags from the given flag set to the given | ||
// configuration. Fields for which the corresponding flag was not used are left | ||
// unmodified. | ||
func SetFlags(fs *pflag.FlagSet, c *LoggingConfiguration) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps
CopyFlags(from *pflag.FlagSet, to *LoggingConfiguration)
?
OverrideLogsFromFlags(from *pflag.FlagSet, to *LoggingConfiguration) ?
var cloneFS pflag.FlagSet | ||
addFlags(c, &cloneFS) | ||
var err error | ||
cloneFS.VisitAll(func (f *pflag.Flag) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cloneFS.VisitAll(func (f *pflag.Flag) { | |
cloneFS.VisitAll(func(f *pflag.Flag) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For --vmodule
, the flag doesn't override: instead, the entries from the config and the command line get merged, with command line entries at the beginning where they have priority. I think that's the right semantic.
But therefore "override" isn't quite appropriate, so I'm going with "copy".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, got it.
/test pull-kubernetes-e2e-kind |
I did some tests based on your changes, It works well. Maybe we need some comments from aojea, as mentioned by #119864 (comment). |
1e85536
to
05f0c50
Compare
@aojea @cyclinder is this PR should be labelled with v1.28 milestone? |
When parsing a config file, all settings derived from command line flags are discarded because only the config settings are used. That has been the traditional behavior for non-logging flags. But `--config ... -v=4` used to work until 71ef0da added logging to the configuration. To restore the original behavior, kube-proxy now: - parses flags - reads the config file - applies logging settings from the flags to the config loaded from file - uses that merged config
05f0c50
to
6ddcdef
Compare
// unmodified. For fields that have multiple values (like vmodule), the values from | ||
// the flags get joined so that the command line flags have priority. | ||
// | ||
// TODO (pohly): move this to logsapi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moving this after the 1.28 release?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, it has to be later
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's to it in 1.29.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can it be moved now? I can work on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, and please do.
Some points to consider:
- Can the same be done for apps using just plain Go flags? The "flag was set" field only exists for pflag. A potential solution might be to wrap the values in a flag.Value implementation which tracks whether
Set
was called. - Examples and docs...
- The original plan was to have a JSON or YAML config that gets decoded into the LoggingConfiguration instance after parsing, in which case this new function isn't needed. It needs to be checked that this works and be documented.
- For configs where that doesn't work (i.e. anything based on registered types via apimachinery like this config here), the new functions need to be documented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A potential solution might be to wrap the values in a flag.Value implementation which tracks whether Set was called.
I found it hard to check if a go flag was set, I tried to check it by using the following codes, but it didn't work.
func isFlagSet(name string) bool {
found := false
flag.Visit(func(f *flag.Flag) {
if f.Name == name {
found = true
}
})
return found
}
Could you give me an example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and we may need a new function called copyLogsFromGoFlags
for this, It may not be for Kubernetes users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A wrapper as in this example is one possible solution:
package main
import (
"flag"
"fmt"
)
type flagWrapper struct {
flag.Value
isSet bool
}
func (f *flagWrapper) Set(value string) error {
f.isSet = true
return f.Value.Set(value)
}
var _ flag.Value = &flagWrapper{}
func main() {
var fs flag.FlagSet
fs.Int("some-int", 42, "the answer")
arg := fs.Lookup("some-int")
wrapper := &flagWrapper{Value: arg.Value}
flag.CommandLine.Var(wrapper, arg.Name, arg.Usage)
flag.Parse()
fmt.Printf("some-int set = %v\n", wrapper.isSet)
}
The downside is that -h
doesn't know about the actual type of the parameter:
$ go run . -h
Usage of /tmp/go-build544671316/b001/exe/testflag:
-some-int value
the answer
Without the wrapper, int
would be shown instead of value
. That's because flag
looks at the type of a value and has special support for builtin value types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the example.
The downside is that -h doesn't know about the actual type of the parameter
Can we explain in the usage what the actual type of the parameter is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Showing "value" isn't a showstopper.
/milestone v1.28 |
@furkatgofurov7 I think yes, but need ack for @pohly @aojea |
// command line flags have priority). Otherwise `--config | ||
// ... -v=5` doesn't work (config resets verbosity even | ||
// when it contains no logging settings). | ||
copyLogsFromFlags(fs, &c.Logging) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this scopes the change to the Logging configuration that makes this change contained
+1
@@ -235,7 +235,7 @@ func NewOptions() *Options { | |||
} | |||
|
|||
// Complete completes all the required options. | |||
func (o *Options) Complete() error { | |||
func (o *Options) Complete(fs *pflag.FlagSet) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we make the Flags part of the Options struct instead of plumbing it down?
I see kube-scheduler has them there too, it feels like is cleaner
kubernetes/cmd/kube-scheduler/app/options/options.go
Lines 75 to 76 in a287fd6
// Flags hold the parsed CLI flags. | |
Flags *cliflag.NamedFlagSets |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps, but that would be a larger change that I don't want to do this close to a release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sgtm
logsapi.AddFlags(to, &cloneFS) | ||
vmodule := to.VModule | ||
to.VModule = nil | ||
var err error | ||
cloneFS.VisitAll(func(f *pflag.Flag) { | ||
if err != nil { | ||
return | ||
} | ||
fsFlag := from.Lookup(f.Name) | ||
if fsFlag == nil { | ||
err = fmt.Errorf("logging flag %s not found in flag set", f.Name) | ||
return | ||
} | ||
if !fsFlag.Changed { | ||
return | ||
} | ||
if setErr := f.Value.Set(fsFlag.Value.String()); setErr != nil { | ||
err = fmt.Errorf("copying flag %s value: %v", f.Name, setErr) | ||
return | ||
} | ||
}) | ||
to.VModule = append(to.VModule, vmodule...) | ||
return err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't really evaluate this logic, I trust other reviewers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand it correctly it overrides everything in logging present in flags, except the to.Vmodule that gets appended
/approve one question about the code structure #119867 (comment), if we make flags part of the I trust @cyclinder or @liggitt and the unit test for this #119867 (comment) @pohly can you remove the WIP from the title if this is ready /assign @liggitt |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aojea, pohly The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Also tagging relevant SIG based on the label on the PR @kubernetes/sig-network-leads |
I'm approving for sig-network Lines 255 to 259 in a287fd6
|
/lgtm I think this is ok, the change is very contained and well covered with unit tests, and we can see also in CI is working fine |
LGTM label has been added. Git tree hash: c0b3740e5310a03e449b9dd9bd5dbc0dd45b0794
|
What type of PR is this?
/kind bug
/kind regression
What this PR does / why we need it:
When parsing a config file, all settings derived from command line flags are discarded because only the config settings are used. That has been the traditional behavior for non-logging flags.
But
--config ... -v=4
used to work until71ef0da added logging to the configuration. To restore the original behavior, kube-proxy now:
Which issue(s) this PR fixes:
Fixes #119864
Special notes for your reviewer:
Needs more tests...
Does this PR introduce a user-facing change?