Skip to content

Vulkan: Select device with env variable, and skip initialize for unused devices. #629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Vulkan: Select device with env variable, and skip initialize for unused devices. #629

wants to merge 2 commits into from

Conversation

stduhpf
Copy link
Contributor

@stduhpf stduhpf commented Mar 14, 2025 •
edited
Loading

Introduces the SD_VK_DEVICE env variable that can be used to chose the device to run inference on.

Also backend was initialized on every device for no apparent reason.

I know it was already possible to hide some devices from the Vulkan backend by setting the GGML_VK_VISIBLE_DEVICES env variable, but this isn't ideal, as it affects every program that uses Vulkan GGML.

(Maybe the same should be done for CUDA and SYCL backends)

@stduhpf stduhpf mentioned this pull request Apr 6, 2025
Closed
@wbruna
Copy link
Contributor

wbruna commented Jun 19, 2025

Works for me... kind of 🙂 I don't think SD_VK_DEVICE can be a replacement for GGML_VK_VISIBLE_DEVICES:

SD_VK_DEVICE=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[WARN ] stable-diffusion.cpp:191  - Cannot find targeted vulkan device (1). Falling back to device 0.
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

GGML_VK_VISIBLE_DEVICES=0 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

GGML_VK_VISIBLE_DEVICES=1 ./sd (...)
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 0

But it's still useful when used together:

GGML_VK_VISIBLE_DEVICES=0,1 SD_VK_DEVICE=1 ./sd (...)
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat
[INFO ] stable-diffusion.cpp:195  - Vulkan: Using device 1

As I understand it, ggml excludes the iGPU by default, so it doesn't even show up on the device list unless it's explicitly enabled.

(aside: I really wish there was a way to specify the devices by a stable ID. Notice how the '0' device points to different GPUs depending on the context...)

@wbruna wbruna mentioned this pull request Jun 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@stduhpf @wbruna