Skip to content

[WIP] HPA - pod selection strategy #132018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 53 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
adc79ab
[WIP] HPA - pod selection strategy
omerap12 May 29, 2025
2b8c5ee
Change to SelectionStrategy
omerap12 May 31, 2025
9490b89
Add converstions
omerap12 May 31, 2025
46b9688
Add converstions
omerap12 May 31, 2025
a94de2a
Fixed bug
omerap12 May 31, 2025
3fea554
mute unit tests for now
omerap12 May 31, 2025
863d68f
implement all functions
omerap12 Jun 1, 2025
0806970
Add unit tests for pod_filters
omerap12 Jun 1, 2025
9f49bf7
Add feature gate
omerap12 Jun 1, 2025
4a1d70d
Add feature gate
omerap12 Jun 1, 2025
579fce5
Add e2e tests
omerap12 Jun 1, 2025
1adaa75
make update
omerap12 Jun 1, 2025
9187143
added adrian to kep
omerap12 Jun 1, 2025
4e7daf0
fallback error
omerap12 Jun 1, 2025
d84a4dd
added comment
omerap12 Jun 1, 2025
3a3f484
Run make update
omerap12 Jun 1, 2025
3e38aa3
Allow HPA controller access to apps resources
adrianmoisey Jun 1, 2025
4934120
Update downgrade feature gate
omerap12 Jun 1, 2025
a689afa
Add validation for HPA's SelectionStrategy
adrianmoisey Jun 1, 2025
fa010f8
Remove unused DefaultSelectionStrategy
adrianmoisey Jun 1, 2025
ea29cfa
Mark as Alpha
adrianmoisey Jun 1, 2025
8db744b
s/HPAselectionStrategy/HPASelectionStrategy/
adrianmoisey Jun 1, 2025
a341d02
make update
omerap12 Jun 1, 2025
33e91cc
fixed unit-tests
omerap12 Jun 2, 2025
eca0cb0
fixed e2e tests
omerap12 Jun 3, 2025
f2336c6
Move podFilterForHpa() inside the computeStatusForPodsMetric() function
adrianmoisey Jun 4, 2025
387fbbb
Add caching to controllers
omerap12 Jun 5, 2025
df85700
Add podFilter
adrianmoisey Jun 5, 2025
e300e07
use dynamic client
omerap12 Jun 12, 2025
d7c01bd
remove app client
omerap12 Jun 12, 2025
3d09f0b
removed error logs
omerap12 Jun 14, 2025
32324e0
moved dynamic client to interface
omerap12 Jun 15, 2025
a3bdb62
Add tests for SelectionStrategy strategy
adrianmoisey Jun 19, 2025
4ac58b5
Fix test when updating status
adrianmoisey Jun 20, 2025
fce1094
Merge branch 'master' into hpaPodSelectionStrategy
adrianmoisey Jun 20, 2025
c2cc0f8
fix test
adrianmoisey Jun 20, 2025
5edfe9b
Should be false
adrianmoisey Jun 21, 2025
76a495e
Add test for TestPrepareForUpdateSelectionStrategyEnabled func
omerap12 Jun 21, 2025
c31ff3f
Add tests for pod_filters and cache
omerap12 Jun 21, 2025
bf26585
Add metrics for cache hit & miss
omerap12 Jun 21, 2025
37f4a92
Add mock monitor
omerap12 Jun 21, 2025
161ce2d
Fix test
adrianmoisey Jun 21, 2025
0796eef
Remove strategy status
omerap12 Jun 22, 2025
ba2d026
Revert "Remove strategy status"
omerap12 Jun 22, 2025
306c4ce
Removed CurrentSelectionStrategy from HPA status
omerap12 Jun 22, 2025
1eaa40e
Update SelectionStrategy description
adrianmoisey Jun 22, 2025
a83fa6e
Handle v1 round-tripping
adrianmoisey Jun 23, 2025
0b429b9
Remove status and handle v2beta1 round tripping
adrianmoisey Jun 24, 2025
536065b
Add unit tests for replica_calculator
omerap12 Jun 27, 2025
a74c3e3
Remove roundtripping for v2beta1
adrianmoisey Jun 30, 2025
061de86
Remove v2beta2 round tripping
adrianmoisey Jun 30, 2025
12a8c43
Revert "Remove v2beta2 round tripping"
adrianmoisey Jun 30, 2025
d614871
Merge branch 'master' into hpaPodSelectionStrategy
omerap12 Jun 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions api/openapi-spec/swagger.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions api/openapi-spec/v3/apis__autoscaling__v2_openapi.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions cmd/kube-controller-manager/app/autoscaling.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,10 @@ func startHPAControllerWithMetricsClient(ctx context.Context, controllerContext
if err != nil {
return nil, false, err
}
dynamicClient, err := dynamic.NewForConfig(hpaClientConfig)
if err != nil {
return nil, false, err
}

go podautoscaler.NewHorizontalController(
ctx,
Expand All @@ -90,6 +94,7 @@ func startHPAControllerWithMetricsClient(ctx context.Context, controllerContext
controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerTolerance,
controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerCPUInitializationPeriod.Duration,
controllerContext.ComponentConfig.HPAController.HorizontalPodAutoscalerInitialReadinessDelay.Duration,
dynamicClient,
).Run(ctx, int(controllerContext.ComponentConfig.HPAController.ConcurrentHorizontalPodAutoscalerSyncs))
return nil, true, nil
}
4 changes: 4 additions & 0 deletions pkg/apis/autoscaling/annotations.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,7 @@ const ToleranceScaleDownAnnotation = "autoscaling.alpha.kubernetes.io/scale-down
// ToleranceScaleUpAnnotation is the annotation which holds the HPA tolerance specs
// when converting the `ScaleUp.Tolerance` field from autoscaling/v2
const ToleranceScaleUpAnnotation = "autoscaling.alpha.kubernetes.io/scale-up-tolerance"

// SelectionStrategyAnnotation is the annotation which holds the pod selection strategy
// when converting the `PodSelectionStrategy` field from autoscaling/v2
const SelectionStrategyAnnotation = "autoscaling.alpha.kubernetes.io/pod-selection-strategy"
9 changes: 8 additions & 1 deletion pkg/apis/autoscaling/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,23 @@ func DropRoundTripHorizontalPodAutoscalerAnnotations(in map[string]string) (out
_, hasToleranceScaleUp := in[ToleranceScaleUpAnnotation]
_, hasMetricsStatuses := in[MetricStatusesAnnotation]
_, hasConditions := in[HorizontalPodAutoscalerConditionsAnnotation]
if hasMetricsSpecs || hasBehaviorSpecs || hasToleranceScaleDown || hasToleranceScaleUp || hasMetricsStatuses || hasConditions {
_, hasSelectionStrategy := in[SelectionStrategyAnnotation]

if hasMetricsSpecs || hasBehaviorSpecs || hasToleranceScaleDown ||
hasToleranceScaleUp || hasMetricsStatuses || hasConditions ||
hasSelectionStrategy {

out = DeepCopyStringMap(in)
delete(out, MetricSpecsAnnotation)
delete(out, BehaviorSpecsAnnotation)
delete(out, ToleranceScaleDownAnnotation)
delete(out, ToleranceScaleUpAnnotation)
delete(out, MetricStatusesAnnotation)
delete(out, HorizontalPodAutoscalerConditionsAnnotation)
delete(out, SelectionStrategyAnnotation)
return out, true
}

return in, false
}

Expand Down
30 changes: 30 additions & 0 deletions pkg/apis/autoscaling/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,17 @@ type CrossVersionObjectReference struct {
APIVersion string
}

// SelectionStrategy defines how pods are selected for metrics collection and scaling decisions
type SelectionStrategy string

const (
// LabelSelector selects pods based on the label selector specified in the HPA's scale target
LabelSelector SelectionStrategy = "LabelSelector"

// OwnerReferences selects pods based on the ownership chain to the HPA's scale target
OwnerReferences SelectionStrategy = "OwnerReferences"
)

// HorizontalPodAutoscalerSpec describes the desired functionality of the HorizontalPodAutoscaler.
type HorizontalPodAutoscalerSpec struct {
// scaleTargetRef points to the target resource to scale, and is used to the pods for which metrics
Expand Down Expand Up @@ -106,6 +117,25 @@ type HorizontalPodAutoscalerSpec struct {
// If not set, the default HPAScalingRules for scale up and scale down are used.
// +optional
Behavior *HorizontalPodAutoscalerBehavior

// SelectionStrategy determines how pods are selected for metrics collection
// and scaling decisions. Valid values are:
// - "LabelSelector": uses the label selector from the scale target (default)
// - "OwnerReferences": only considers pods owned by the scale target
//
// If not set, the default value "LabelSelector" is used, which maintains
// backward compatibility with existing behavior.
//
// For example, when using "OwnerReferences" with a Deployment target,
// only pods directly owned by the Deployment's ReplicaSets will be considered,
// even if other pods match the label selector.
//
// This is an alpha field and requires enabling the HPASelectionStrategy
// feature gate.
//
// +featureGate=HPASelectionStrategy
// +optional
SelectionStrategy *SelectionStrategy
}

// HorizontalPodAutoscalerBehavior configures a scaling behavior for Up and Down direction
Expand Down
13 changes: 13 additions & 0 deletions pkg/apis/autoscaling/v1/conversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,19 @@ func Convert_autoscaling_HorizontalPodAutoscaler_To_v1_HorizontalPodAutoscaler(i
out.Annotations[autoscaling.BehaviorSpecsAnnotation] = string(behaviorEnc)
}

if in.Spec.SelectionStrategy != nil {
selectionStrategyEnc, err := json.Marshal(in.Spec.SelectionStrategy)
if err != nil {
return err
}
// copy before mutating
if !copiedAnnotations {
copiedAnnotations = true
out.Annotations = autoscaling.DeepCopyStringMap(out.Annotations)
}
out.Annotations[autoscaling.SelectionStrategyAnnotation] = string(selectionStrategyEnc)
}

if len(in.Status.Conditions) > 0 {
currentConditionsEnc, err := json.Marshal(currentConditions)
if err != nil {
Expand Down
1 change: 1 addition & 0 deletions pkg/apis/autoscaling/v1/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions pkg/apis/autoscaling/v2/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pkg/apis/autoscaling/v2beta1/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions pkg/apis/autoscaling/v2beta2/conversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ func Convert_autoscaling_HorizontalPodAutoscaler_To_v2beta2_HorizontalPodAutosca
annotations, copiedAnnotations := autoscaling.DropRoundTripHorizontalPodAutoscalerAnnotations(out.Annotations)
out.Annotations = annotations

if in.Spec.SelectionStrategy != nil {
if !copiedAnnotations {
copiedAnnotations = true
out.Annotations = autoscaling.DeepCopyStringMap(out.Annotations)
}
out.Annotations[autoscaling.SelectionStrategyAnnotation] = string(*in.Spec.SelectionStrategy)
}

behavior := in.Spec.Behavior
if behavior == nil {
return nil
Expand Down Expand Up @@ -87,6 +95,11 @@ func Convert_v2beta2_HorizontalPodAutoscaler_To_autoscaling_HorizontalPodAutosca
}
out.Spec.Behavior.ScaleUp.Tolerance = &q
}

if strategyStr, ok := out.Annotations[autoscaling.SelectionStrategyAnnotation]; ok {
strategy := autoscaling.SelectionStrategy(strategyStr)
out.Spec.SelectionStrategy = &strategy
}
// Do not save round-trip annotations in internal resource
out.Annotations, _ = autoscaling.DropRoundTripHorizontalPodAutoscalerAnnotations(out.Annotations)
return nil
Expand All @@ -101,3 +114,19 @@ func Convert_autoscaling_HPAScalingRules_To_v2beta2_HPAScalingRules(in *autoscal
// Tolerance field is handled in the HorizontalPodAutoscaler conversion function.
return autoConvert_autoscaling_HPAScalingRules_To_v2beta2_HPAScalingRules(in, out, s)
}

func Convert_autoscaling_HorizontalPodAutoscalerSpec_To_v2beta2_HorizontalPodAutoscalerSpec(in *autoscaling.HorizontalPodAutoscalerSpec, out *autoscalingv2beta2.HorizontalPodAutoscalerSpec, s conversion.Scope) error {
if err := autoConvert_autoscaling_HorizontalPodAutoscalerSpec_To_v2beta2_HorizontalPodAutoscalerSpec(in, out, s); err != nil {
return err
}
// SelectionStrategy is handled in the HorizontalPodAutoscaler conversion function
return nil
}

func Convert_autoscaling_HorizontalPodAutoscalerStatus_To_v2beta2_HorizontalPodAutoscalerStatus(in *autoscaling.HorizontalPodAutoscalerStatus, out *autoscalingv2beta2.HorizontalPodAutoscalerStatus, s conversion.Scope) error {
if err := autoConvert_autoscaling_HorizontalPodAutoscalerStatus_To_v2beta2_HorizontalPodAutoscalerStatus(in, out, s); err != nil {
return err
}
// CurrentSelectionStrategy is handled in the HorizontalPodAutoscaler conversion function
return nil
}
31 changes: 11 additions & 20 deletions pkg/apis/autoscaling/v2beta2/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

23 changes: 23 additions & 0 deletions pkg/apis/autoscaling/validation/validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package validation

import (
"fmt"
"slices"

apimachineryvalidation "k8s.io/apimachinery/pkg/api/validation"
pathvalidation "k8s.io/apimachinery/pkg/api/validation/path"
Expand Down Expand Up @@ -73,6 +74,10 @@ func validateHorizontalPodAutoscalerSpec(autoscaler autoscaling.HorizontalPodAut
if refErrs := validateBehavior(autoscaler.Behavior, fldPath.Child("behavior"), opts); len(refErrs) > 0 {
allErrs = append(allErrs, refErrs...)
}
if refErrs := valdidateSelectionStrategy(autoscaler.SelectionStrategy, fldPath.Child("selectionStrategy"), opts); len(refErrs) > 0 {
allErrs = append(allErrs, refErrs...)
}

return allErrs
}

Expand Down Expand Up @@ -172,6 +177,24 @@ func validateBehavior(behavior *autoscaling.HorizontalPodAutoscalerBehavior, fld
return allErrs
}

func valdidateSelectionStrategy(selectionStrategy *autoscaling.SelectionStrategy, fldPath *field.Path, opts HorizontalPodAutoscalerSpecValidationOptions) field.ErrorList {
allErrs := field.ErrorList{}
if selectionStrategy == nil {
return allErrs
}

supportedSelectionStrategy := []autoscaling.SelectionStrategy{
autoscaling.LabelSelector,
autoscaling.OwnerReferences,
}

if !slices.Contains(supportedSelectionStrategy, *selectionStrategy) {
allErrs = append(allErrs, field.NotSupported(field.NewPath("spec").Child("selectionStrategy"), *selectionStrategy, supportedSelectionStrategy))
}

return allErrs
}

var validSelectPolicyTypes = sets.NewString(string(autoscaling.MaxPolicySelect), string(autoscaling.MinPolicySelect), string(autoscaling.DisabledPolicySelect))
var validSelectPolicyTypesList = validSelectPolicyTypes.List()

Expand Down
5 changes: 5 additions & 0 deletions pkg/apis/autoscaling/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading