The problem with this approach will appear when a second bunch of pods will start to be created. Then, we would end up with conflict on creating a pod in a place of a pod that wasn't deleted.

Maybe you could try waiting for all pods to be scheduled just after all of them were created, before deleting any?

Doing this, we will wait only for the first bunch of pods. Then, after we create a second bunch, we will proceed with deletion without waiting.

@macsko
Thank you for pointing out this issue! You're right that the current implementation only waits for the first batch of pods; subsequent batches proceed without waiting.

I've been thinking about how to properly address this while maintaining the intended behavior of churnOp, and here are a few approaches I'm considering:

Option 1: Keep it simple — remove the waiting logic entirely.

• Rely only on the scheduled check before deletion.
• Unscheduled pods will remain until they are scheduled.
  Pros: Simple, consistent behavior throughout the test
  Cons: Some pods might accumulate in the pending state.

Option 2: Periodic synchronization - Wait every N cycles

  // Wait when starting a new cycle of pods
  if hasPods && count % op.Number == 0 && count > op.Number {
      waitForScheduledPodsInNamespace(...)
  }

• Pros: Ensures fair treatment of all pod generations
• Cons: Adds complexity and periodic pauses

Option 3: Dynamic Throttling — Adjusts the churn rate based on the scheduled pod ratio.
Only proceed with deletion when X% of pods are scheduled.

• Self-adjusts to scheduler performance
  Pros: Adaptive to system state
  Cons: More complex implementation

I'm leaning towards option 2, which removes the wait entirely. This option keeps the implementation simple while still addressing the original issue, #125974, by preventing the deletion of unscheduled pods.

What's your preference? Are there any other approaches that would better align with the goals of scheduler_perf testing?

What if we do:

		churnFns = append(churnFns, func(name string) string {
			if name != "" {
				// New: Wait for `name` pod to be scheduled here, before deleting
 				if isPod(name) {
 					waitForPod(name)
 				}
				if err := dynRes.Delete(e.tCtx, name, metav1.DeleteOptions{}); err != nil && !errors.Is(err, context.Canceled) {
					e.tCtx.Errorf("op %d: unable to delete %v: %v", opIndex, name, err)
				}
				return ""
			}

			live, err := dynRes.Create(e.tCtx, unstructuredObj, metav1.CreateOptions{})
			if err != nil {
				return ""
			}
			return live.GetName()
		})

We could also make this waiting before deletion mechanism under boolean flag to allow to use the old way as well.