Description
What happened?
When an HPA targets a Deployment which has a label selector matching Pods that don't belong to it (overlapping labels, for example), those "other" Pods are considered by the HPA to be part of the targeted HPA.
What did you expect to happen?
I have always been lead to believe that this behaviour is correct and the user should have labels that are specific enough to not overlap other Deployments.
This is stated here:
Do not overlap labels or selectors with other controllers (including other Deployments and StatefulSets). Kubernetes doesn't stop you from overlapping, and if multiple controllers have overlapping selectors those controllers might conflict and behave unexpectedly.
However, recently the VPA fixed the same behaviour in kubernetes/autoscaler#6460
Which makes me wonder if I should expect the HPA to test the owner reference too, and only target that Deployment's pods.
How can we reproduce it (as minimally and precisely as possible)?
- Create 2 deployments, both with the same labels/selectors
- In 1 Deployment set resources
- Deploy an HPA (and metrics-server)
- Watch the events of the HPA throw error events related to Pods not under control of the HPA's deployment
- ie:
Warning FailedGetResourceMetric 87s (x3 over 117s) horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu in container ubuntu of Pod other-64886557cb-ldtnt
- ie:
YAML files
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: workload
name: workload
spec:
replicas: 1
selector:
matchLabels:
app: other
template:
metadata:
labels:
app: other
spec:
containers:
- command:
- /bin/bash
- -c
- sha256sum /dev/zero
image: ubuntu
imagePullPolicy: Always
name: nginx
resources:
requests:
cpu: 10m
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: other
name: other
spec:
replicas: 1
selector:
matchLabels:
app: other
template:
metadata:
labels:
app: other
spec:
containers:
- command:
- /bin/bash
- -c
- sleep infinity
image: ubuntu
imagePullPolicy: Always
name: ubuntu
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: workload
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: workload
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2
Cloud provider
- kind
- GKE
- kOps (AWS)
OS version
No response