Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e pods: fix WaitForPodsResponding retry #120559

Conversation

pohly
Copy link
Contributor

@pohly pohly commented Sep 11, 2023

What type of PR is this?

/kind bug
/kind failing-test

What this PR does / why we need it:

The status error was embedded inside the new error constructed by WaitForPodsResponding's get function, but not wrapped. Therefore apierrors.IsServiceUnavailable(err) didn't find it and returned false -> no retries.

Wrapping fixes this and Gomega formatting of the error remains useful:

err := &errors.StatusError{}
err.ErrStatus.Code = 503
err.ErrStatus.Message = "temporary failure"

err2 := fmt.Errorf("Controller %s: failed to Get from replica pod %s:\n%w\nPod status:\n%s",
	"foo", "bar",
	err, "some status")
fmt.Println(format.Object(err2, 1))
    fmt.Println(errors.IsServiceUnavailable(err2))

=>

<*fmt.wrapError | 0xc000139340>:
Controller foo: failed to Get from replica pod bar:
temporary failure
Pod status:
some status
{
    msg: "Controller foo: failed to Get from replica pod bar:\ntemporary failure\nPod status:\nsome status",
    err: <*errors.StatusError | 0xc0001a01e0>{
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "",
            Message: "temporary failure",
            Reason: "",
            Details: nil,
            Code: 503,
        },
    },
}

true

Which issue(s) this PR fixes:

Fixes #120539

Special notes for your reviewer:

Does this PR introduce a user-facing change?

e2e framework: retrying after intermittent apiserver failures was fixed in WaitForPodsResponding

/cc @alculquicondor

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 11, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Sep 11, 2023
@k8s-ci-robot k8s-ci-robot added area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 11, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 11, 2023
controllerName, pod.Name,
format.Object(err, 1), format.Object(pod.Status, 1))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, very subtle, impressive

@aojea
Copy link
Member

aojea commented Sep 11, 2023

/lgtm

Thanks, can we backport it?

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 11, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 126881417fbd81f2b74c011938ee427b5671acaa

@aojea
Copy link
Member

aojea commented Sep 11, 2023

linter complain

ERROR: test/e2e/framework/pod/wait.go:586:17: ST1005: error strings should not be capitalized (stylecheck)
ERROR: return nil, fmt.Errorf("Controller %s: failed to Get from replica pod %s:\n%w\nPod status:\n%s",

The status error was embedded inside the new error constructed by
WaitForPodsResponding's get function, but not wrapped. Therefore
`apierrors.IsServiceUnavailable(err)` didn't find it and returned false -> no
retries.

Wrapping fixes this and Gomega formatting of the error remains useful:

	err := &errors.StatusError{}
	err.ErrStatus.Code = 503
	err.ErrStatus.Message = "temporary failure"

	err2 := fmt.Errorf("Controller %s: failed to Get from replica pod %s:\n%w\nPod status:\n%s",
		"foo", "bar",
		err, "some status")
	fmt.Println(format.Object(err2, 1))
        fmt.Println(errors.IsServiceUnavailable(err2))

=>

    <*fmt.wrapError | 0xc000139340>:
    Controller foo: failed to Get from replica pod bar:
    temporary failure
    Pod status:
    some status
    {
        msg: "Controller foo: failed to Get from replica pod bar:\ntemporary failure\nPod status:\nsome status",
        err: <*errors.StatusError | 0xc0001a01e0>{
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "",
                Message: "temporary failure",
                Reason: "",
                Details: nil,
                Code: 503,
            },
        },
    }

    true
@pohly pohly force-pushed the e2e-framework-WaitForPodsResponding-retry branch from 9fa43d3 to fc3ee07 Compare September 11, 2023 09:54
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 11, 2023
@pohly
Copy link
Contributor Author

pohly commented Sep 11, 2023

linter complain

Fixed.

Thanks, can we backport it?

I agree, we should. Let's merge it and I'll prepare cherry-picks.

@aojea
Copy link
Member

aojea commented Sep 11, 2023

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 11, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 79293d1c1c54b9f4c2f686cf254949b4d8b43293

@pohly
Copy link
Contributor Author

pohly commented Sep 11, 2023

/retest

1 similar comment
@pohly
Copy link
Contributor Author

pohly commented Sep 11, 2023

/retest

@k8s-ci-robot k8s-ci-robot merged commit b1161a8 into kubernetes:master Sep 11, 2023
15 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Sep 11, 2023
@pohly
Copy link
Contributor Author

pohly commented Sep 11, 2023

k8s-ci-robot added a commit that referenced this pull request Sep 14, 2023
…9-origin-release-1.27

Automated cherry pick of #120559: e2e pods: fix WaitForPodsResponding retry
k8s-ci-robot added a commit that referenced this pull request Sep 14, 2023
…9-origin-release-1.28

Automated cherry pick of #120559: e2e pods: fix WaitForPodsResponding retry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

e2e WaitForPodsResponding does not retry
3 participants