Skip to content

Overlapping labels can lead to HPA matching incorrect pods #124307

Closed as not planned
@adrianmoisey

Description

@adrianmoisey

What happened?

When an HPA targets a Deployment which has a label selector matching Pods that don't belong to it (overlapping labels, for example), those "other" Pods are considered by the HPA to be part of the targeted HPA.

What did you expect to happen?

I have always been lead to believe that this behaviour is correct and the user should have labels that are specific enough to not overlap other Deployments.
This is stated here:

Do not overlap labels or selectors with other controllers (including other Deployments and StatefulSets). Kubernetes doesn't stop you from overlapping, and if multiple controllers have overlapping selectors those controllers might conflict and behave unexpectedly.

However, recently the VPA fixed the same behaviour in kubernetes/autoscaler#6460

Which makes me wonder if I should expect the HPA to test the owner reference too, and only target that Deployment's pods.

How can we reproduce it (as minimally and precisely as possible)?

  1. Create 2 deployments, both with the same labels/selectors
  2. In 1 Deployment set resources
  3. Deploy an HPA (and metrics-server)
  4. Watch the events of the HPA throw error events related to Pods not under control of the HPA's deployment
    1. ie: Warning FailedGetResourceMetric 87s (x3 over 117s) horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu in container ubuntu of Pod other-64886557cb-ldtnt
YAML files
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
  app: workload
name: workload
spec:
replicas: 1
selector:
  matchLabels:
    app: other
template:
  metadata:
    labels:
      app: other
  spec:
    containers:
    - command:
      - /bin/bash
      - -c
      - sha256sum /dev/zero
      image: ubuntu
      imagePullPolicy: Always
      name: nginx
      resources:
        requests:
          cpu: 10m
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
  app: other
name: other
spec:
replicas: 1
selector:
  matchLabels:
    app: other
template:
  metadata:
    labels:
      app: other
  spec:
    containers:
    - command:
      - /bin/bash
      - -c
      - sleep infinity
      image: ubuntu
      imagePullPolicy: Always
      name: ubuntu
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: workload
spec:
scaleTargetRef:
  apiVersion: apps/v1
  kind: Deployment
  name: workload
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 50

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

Cloud provider

  • kind
  • GKE
  • kOps (AWS)

OS version

No response

Install tools

kind

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/autoscalingCategorizes an issue or PR as relevant to SIG Autoscaling.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions