Closed
Description
Which jobs are flaking?
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-unit/1933607361415483392
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-unit-1-33/1931163207162925056
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-unit-ppc64le/1935867526592335872
Which tests are flaking?
k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go: TestResourceManagerAddedCRD
k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:TestDiscoveryControllerResourceManagerRemovedCRD
Since when has it been flaking?
Triage tool shows the earliest failures on 6/13/2025, 7:47:46 AM as per Link.
Testgrid link
https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit
Reason for failure (if possible)
=== RUN TestResourceManagerAddedCRD
customresource_discovery_controller_test.go:326:
Error Trace: /home/prow/go/src/k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:326
Error: Received unexpected error:
timed out waiting for the condition
Test: TestResourceManagerAddedCRD
--- FAIL: TestResourceManagerAddedCRD (1.01s)
=== RUN TestDiscoveryControllerResourceManagerRemovedCRD
customresource_discovery_controller_test.go:408:
Error Trace: /home/prow/go/src/k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:408
Error: Received unexpected error:
timed out waiting for the condition
Test: TestDiscoveryControllerResourceManagerRemovedCRD
--- FAIL: TestDiscoveryControllerResourceManagerRemovedCRD (1.00s)
Anything else we need to know?
We were able to reproduce this flake locally using stress tool.
Steps to reproduce:
- Install stress tool (# go install golang.org/x/tools/cmd/stress@latest).
- Clone the k8s repo (git clone https://github.com/kubernetes/kubernetes.git).
- cd kubernetes
- go test ./staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/ -c race
- /root/go/bin/stress ./apiserver.test -test.run TestResourceManagerAddedCRD
[root@raji-x86-workspace1 kubernetes]# uname -a
Linux raji-x86-workspace1.fyre.ibm.com 5.14.0-573.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 18 15:33:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
[root@raji-x86-workspace1 kubernetes]#
[root@raji-x86-workspace1 kubernetes]# /root/go/bin/stress ./apiserver.test -test.run TestResourceManagerAddedCRD
5s: 297 runs so far, 0 failures, 16 active
10s: 614 runs so far, 0 failures, 16 active
/tmp/go-stress-20250620T013232-1920411983
I0620 01:32:42.568589 872999 customresource_discovery_controller.go:294] Starting DiscoveryController
--- FAIL: TestResourceManagerAddedCRD (1.00s)
customresource_discovery_controller_test.go:326:
Error Trace: /root/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:326
Error: Received unexpected error:
timed out waiting for the condition
Test: TestResourceManagerAddedCRD
FAIL
I0620 01:32:43.569575 872999 customresource_discovery_controller.go:332] Shutting down DiscoveryController
ERROR: exit status 1
15s: 922 runs so far, 1 failures (0.11%), 16 active
/tmp/go-stress-20250620T013232-2803581754
I0620 01:32:47.777126 877060 customresource_discovery_controller.go:294] Starting DiscoveryController
--- FAIL: TestResourceManagerAddedCRD (1.00s)
customresource_discovery_controller_test.go:326:
Error Trace: /root/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:326
Error: Received unexpected error:
timed out waiting for the condition
Test: TestResourceManagerAddedCRD
FAIL
I0620 01:32:48.779243 877060 customresource_discovery_controller.go:332] Shutting down DiscoveryController
ERROR: exit status 1
/tmp/go-stress-20250620T013232-3917711053
I0620 01:32:50.684182 879309 customresource_discovery_controller.go:294] Starting DiscoveryController
--- FAIL: TestResourceManagerAddedCRD (1.00s)
customresource_discovery_controller_test.go:326:
Error Trace: /root/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:326
Error: Received unexpected error:
timed out waiting for the condition
Test: TestResourceManagerAddedCRD
FAIL
ERROR: exit status 1
20s: 1228 runs so far, 3 failures (0.24%), 16 active
^C
[root@raji-x86-workspace1 kubernetes]#
[root@raji-x86-workspace1 kubernetes]# /root/go/bin/stress ./apiserver.test -test.run TestDiscoveryControllerResourceManagerRemovedCRD
/tmp/go-stress-20250620T013722-1214723754
I0620 01:37:23.401497 882604 customresource_discovery_controller.go:294] Starting DiscoveryController
--- FAIL: TestDiscoveryControllerResourceManagerRemovedCRD (1.01s)
customresource_discovery_controller_test.go:408:
Error Trace: /root/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:408
Error: Received unexpected error:
timed out waiting for the condition
Test: TestDiscoveryControllerResourceManagerRemovedCRD
FAIL
ERROR: exit status 1
5s: 189 runs so far, 1 failures (0.53%), 16 active
/tmp/go-stress-20250620T013722-3937345706
I0620 01:37:30.666243 886121 customresource_discovery_controller.go:294] Starting DiscoveryController
--- FAIL: TestDiscoveryControllerResourceManagerRemovedCRD (1.00s)
customresource_discovery_controller_test.go:408:
Error Trace: /root/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_discovery_controller_test.go:408
Error: Received unexpected error:
timed out waiting for the condition
Test: TestDiscoveryControllerResourceManagerRemovedCRD
FAIL
ERROR: exit status 1
10s: 398 runs so far, 2 failures (0.50%), 16 active
^C
[root@raji-x86-workspace1 kubernetes]#
Relevant SIG(s)
/sig testing