[WIP] Dra device health status #130606

Jpsassine · 2025-03-06T01:09:57Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements KEP-4680 by adding device health tracking for DRA plugins and integrating this status into the PodStatus. It allows Kubelet and users to observe the health of allocated DRA resources.

Optional gRPC Health Service (dra-health/v1alpha1):
- Defines a new NodeHealthService with a WatchResources stream RPC in staging/src/k8s.io/kubelet/pkg/apis/dra-health/v1alpha1/api.proto.
- DRA plugins can optionally implement this service to stream device health updates (pool_name, device_name, health, Last_updated) to Kubelet.
Kubelet Plugin Integration (cm/dra/plugin):
- RegistrationHandler (registration.go) updated to:
  - Attempt to start the WatchResources stream upon plugin registration via plugin.WatchResources.
  - Handle plugins that don't implement the NodeHealthService (or other stream startup errors) by logging the error and proceeding with registration without health monitoring.
  - Launch a handler goroutine (via StreamHandler interface implemented by dra.ManagerImpl) in the DRA Manager upon successful stream initiation.
  - Cancel the health stream context (HealthStreamCancel) during plugin deregistration (DeregisterPlugin) or replacement.
Health Cache (cm/dra/healthinfo.go, cm/dra/state/state.go):
- Implements healthInfoCache for persistent (dra_health_state file), thread-safe storage of device health (Healthy, Unhealthy, Unknown) and timestamps.
- Includes updateHealthInfo for full-state reconciliation based on plugin updates, handling timeouts (healthTimeout constant, e.g., 30s) by marking stale devices as "Unknown". Saves checkpoint on change.
- Includes getHealthInfo to retrieve current status (returns "Unknown" if stale/missing) and clearDriver for cleanup.
DRA Manager Integration (cm/dra/manager.go):
- ManagerImpl implements the plugin.StreamHandler interface.
- HandleWatchResourcesStream goroutine consumes updates from the plugin stream (NodeHealth.WatchResourcesClient), calls healthInfoCache.updateHealthInfo, finds affected Pod UIDs from claimInfoCache, and sends notifications via an internal update channel (non-blocking).
- Added defer healthInfoCache.clearDriver(pluginName) within HandleWatchResourcesStream to ensure cache cleanup for the driver upon goroutine exit (due to error, cancellation, or EOF).
- Adds Updates() method to return the update channel.
- Updates UpdateAllocatedResourcesStatus to read health from healthInfoCache and populate pod.Status.ContainerStatuses[].AllocatedResourcesStatus using the KEP-specified structure: v1.ResourceStatus (named by claim (Name field), containing a Resources slice where each element is a v1.ResourceHealth struct per device (ResourceID and Health fields)).
Container Manager Integration (cm/container_manager_linux.go):
- Updates() method merges update signals from Device Manager and DRA Manager (via draManager.Updates()).
- UpdateAllocatedResourcesStatus() method calls both Device Manager and DRA Manager update functions (DRA call guarded by feature gate DynamicResourceAllocation and nil check).
Testing:
- Adds unit tests for healthinfo.go (healthinfo_test.go).
- Adds unit tests for manager.go (manager_test.go) covering HandleWatchResourcesStream and UpdateAllocatedResourcesStatus. Fixes existing tests (TestPrepareResources, TestUnprepareResources) to align with updated signatures/logic.
- WIP E2E Test

Which issue(s) this PR fixes:

Fixes #126243

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4680-add-resource-health-to-pod-status/README.md

linux-foundation-easycla · 2025-03-06T01:10:01Z

The committers listed above are authorized under a signed CLA.

✅ login: Jpsassine / name: JP (3ee3e45, 03aae5e, d999489, e538497)

k8s-ci-robot · 2025-03-06T01:10:06Z

Hi @Jpsassine. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-03-06T01:10:06Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

googs1025 · 2025-03-06T02:58:26Z

/ok-to-test

bart0sh · 2025-03-10T09:00:24Z

@Jpsassine Thank you for your PR. Please sign the CLA to proceed further, thanks.

BTW, is there any KEP or another design document describing/discussing these changes? If so, please provide links in the PR description.

Jpsassine · 2025-03-11T17:43:03Z

/easycla

haosdent · 2025-06-21T13:50:38Z

pkg/kubelet/cm/dra/healthinfo.go

+		}
+		return err
+	}
+	return json.Unmarshal(data, cache.HealthInfo)


Should we wrap error with specific message to make it easier to troubleshoot?

pkg/kubelet/cm/dra/healthinfo.go

This commit implements the Pod Resource Health Status feature as described in KEP-4680. This allows Dynamic Resource Allocation (DRA) plugins to report the health of their devices back to the Kubelet. The primary changes include: - A new, optional, server-streaming gRPC service, `v1alpha1.NodeHealth`, is introduced. DRA drivers can implement the `WatchResources` RPC on this service to stream health updates. - The Kubelet's `dra.Manager` is updated to act as a client to this service. It listens for health updates, maintains an internal health cache for all devices, and triggers pod status updates when a device's health changes. - The Pod's `AllocatedResourcesStatus` field is now populated with the health ("Healthy", "Unhealthy", "Unknown") of its assigned devices, making this information visible via the Kubernetes API. Add first e2e test & helper funcs trying to fix pod syncing status & add debug logs

k8s-ci-robot · 2025-07-01T07:03:05Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jpsassine
Once this PR has been reviewed and has the lgtm label, please assign klueska for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/kubelet/OWNERS
staging/src/k8s.io/dynamic-resource-allocation/OWNERS
staging/src/k8s.io/kubelet/OWNERS
test/e2e/dra/OWNERS
test/e2e_node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

current logic Create DeviceClass. Create ResourceClaim (without status). Create Pod. Update ResourceClaim status (too late) Updated logic Create DeviceClass. Create ResourceClaim (without status). Update ResourceClaim status. Wait for the dra.Manager to see the updated claim Create Pod.

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 6, 2025

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 6, 2025

k8s-ci-robot requested review from haircommander and tzneal March 6, 2025 01:10

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. wg/device-management Categorizes an issue or PR as relevant to WG Device Management. labels Mar 6, 2025

github-project-automation bot added this to SIG Node: code and documentation PRs Mar 6, 2025

k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Mar 6, 2025

github-project-automation bot added this to Dynamic Resource Allocation Mar 6, 2025

github-project-automation bot moved this to Triage in SIG Node: code and documentation PRs Mar 6, 2025

github-project-automation bot moved this to 🆕 New in Dynamic Resource Allocation Mar 6, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 6, 2025

bart0sh moved this from Triage to Waiting on Author in SIG Node: code and documentation PRs Mar 10, 2025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 11, 2025

Jpsassine force-pushed the dra_device_health_status branch 9 times, most recently from 206e688 to 87d4a5d Compare June 20, 2025 23:51

haosdent reviewed Jun 21, 2025

View reviewed changes

Jpsassine force-pushed the dra_device_health_status branch 3 times, most recently from 85fc598 to 4512f49 Compare June 26, 2025 03:31

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 26, 2025

Jpsassine added 2 commits July 1, 2025 01:22

Fixing deadlocks and race conditions with unit testing after rebase

e538497

Jpsassine force-pushed the dra_device_health_status branch from 4512f49 to e538497 Compare July 1, 2025 05:51

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 1, 2025

Jpsassine force-pushed the dra_device_health_status branch from 9ac3a2a to 0e72a66 Compare July 1, 2025 07:23

e2e support: detect when a plugin implements NodeHealthServer interface

3ee3e45

Jpsassine force-pushed the dra_device_health_status branch 7 times, most recently from 507632d to 94bc6a5 Compare July 1, 2025 09:32

Jpsassine force-pushed the dra_device_health_status branch from 94bc6a5 to d999489 Compare July 1, 2025 09:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Dra device health status #130606

[WIP] Dra device health status #130606

Uh oh!

Jpsassine commented Mar 6, 2025 •

edited

Loading

Uh oh!

linux-foundation-easycla bot commented Mar 6, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Mar 6, 2025

Uh oh!

k8s-ci-robot commented Mar 6, 2025

Uh oh!

googs1025 commented Mar 6, 2025

Uh oh!

bart0sh commented Mar 10, 2025

Uh oh!

Jpsassine commented Mar 11, 2025

Uh oh!

haosdent Jun 21, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

Uh oh!

[WIP] Dra device health status #130606

Are you sure you want to change the base?

[WIP] Dra device health status #130606

Uh oh!

Conversation

Jpsassine commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

linux-foundation-easycla bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Mar 6, 2025

Uh oh!

k8s-ci-robot commented Mar 6, 2025

Uh oh!

googs1025 commented Mar 6, 2025

Uh oh!

bart0sh commented Mar 10, 2025

Uh oh!

Jpsassine commented Mar 11, 2025

Uh oh!

haosdent Jun 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

Uh oh!

Jpsassine commented Mar 6, 2025 •

edited

Loading

linux-foundation-easycla bot commented Mar 6, 2025 •

edited

Loading