Fix flake caused by invalid detection of active policies in VAP integration tests #132502
Conversation
|
Skipping CI for Draft Pull Request. |
|
/test pull-kubernetes-integration |
|
Increasing the policy refresh interval is enough to consistently reproduce this failure. Because the "test-marker" check is identical across test cases, it can proceed even when the policy from the previous test case is still in effect. Generating a random marker name for each test policy seems to resolve it, even with a very wide policy refresh interval. /test pull-kubernetes-integration |
Writes to policy resources don't instantaneously take effect in admission. ValidatingAdmissionPolicy integration tests determine that the policies under test have taken effect by adding a sentinel policy rule and polling until that rule is applied to a request. If the marker resource names are the same for each test case in a series of test cases, then observing a policy's effect on a marker request only indicates that _any_ test policy is in effect, but it's not necessarily the policy the current test case is waiting for. For example: 1. Test 1 creates a policy and binding. 2. The policy and binding are observed by the admission plugin and take effect. 3. Test 1 observes that a policy is in effect via marker requests. 4. Test 1 exercises the behavior under test and successfully deletes the policy and binding it created. 5. Test 2 creates a policy and binding. 6. Test 2 observes that a policy is in effect via marker requests, but the policy in effect is still the one created by Test 1. 7. Test 2 exercises the behavior under test, which fails because it was evaluated against Test 1's policy. Generating a per-policy name for the marker resource in each test resolves the timing issue. In the example, step (6) will not proceed until the admission plugin has observed the policy and binding created in (5).
8dae9ea to
190c8c7
Compare
|
/cc @jpbetz |
|
The PR job history includes a run where I had changed nothing except increased |
|
@benluddy: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
LGTM label has been added. Git tree hash: 57efaf9f65dafdccca53e39771c5e7d288ceda4f
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: benluddy, liggitt The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/triage accepted |
What type of PR is this?
/kind flake
/sig api-machinery
What this PR does / why we need it:
Writes to policy resources don't instantaneously take effect in admission. ValidatingAdmissionPolicy
integration tests determine that the policies under test have taken effect by adding a sentinel
policy rule and polling until that rule is applied to a request.
If the marker resource names are the same for each test case in a series of test cases, then
observing a policy's effect on a marker request only indicates that any test policy is in effect,
but it's not necessarily the policy the current test case is waiting for. For example:
Test 1 creates a policy and binding.
The policy and binding are observed by the admission plugin and take effect.
Test 1 observes that a policy is in effect via marker requests.
Test 1 exercises the behavior under test and successfully deletes the policy and binding it
created.
Test 2 creates a policy and binding.
Test 2 observes that a policy is in effect via marker requests, but the policy in effect is still
the one created by Test 1.
Test 2 exercises the behavior under test, which fails because it was evaluated against Test 1's
policy.
Generating a per-policy name for the marker resource in each test resolves the timing issue. In the
example, step (6) will not proceed until the admission plugin has observed the policy and binding
created in (5).
Which issue(s) this PR is related to:
N/A
Special notes for your reviewer:
@bertinatto first reported this in https://kubernetes.slack.com/archives/C02TTBG6LF4/p1746818599391839 after seeing this flake in OpenShift CI.
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: