You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
GetNumaNode reads a device's NUMA node from a hardcoded /sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.
This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same --sysfs-root request for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).
Problem
internal/rm/nvml_devices.go:150-162:
busID:=strings.ToLower(strings.TrimPrefix(uint8Slice(info.BusId[:]).String(), "0000"))
b, err:=os.ReadFile(fmt.Sprintf("/sys/bus/pci/devices/%s/numa_node", busID))
iferr!=nil {
returnfalse, 0, nil// error swallowed -> device reported with no NUMA
}
The /sys prefix is a literal; nothing overrides it (--nvidia-driver-root / --nvidia-dev-root feed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails, hasNuma=false, and BuildDevice (internal/rm/devices.go:93-101) leaves Device.Topology nil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.
Proposed solution
Flag --sysfs-root (default /sys), matching #264 and the existing --nvidia-*-root family:
In-tree precedent for the same read: go-nvlib's WithPCIDevicesRoot (pkg/nvpci/nvpci.go:187,317, mock at mock.go:104) and k8s dynamic-resource-allocation's WithFSFromRoot (which #264 builds on). Default unchanged means zero impact on real deployments.
Alternatives
nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returns NVML_ERROR_NOT_SUPPORTED for it, and on real hardware it signals "no NUMA" via return code (not the sysfs -1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway.
Scoped to GetNumaNode (NVML full GPU + MIG, which delegates to the parent). A second hardcoded root at internal/vgpu/pciutil.go:42 (GFD vGPU detection) could adopt the same flag later. Happy to send a PR.
Summary
GetNumaNodereads a device's NUMA node from a hardcoded/sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same
--sysfs-rootrequest for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).Problem
internal/rm/nvml_devices.go:150-162:The
/sysprefix is a literal; nothing overrides it (--nvidia-driver-root/--nvidia-dev-rootfeed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails,hasNuma=false, andBuildDevice(internal/rm/devices.go:93-101) leavesDevice.Topologynil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.Proposed solution
Flag
--sysfs-root(default/sys), matching #264 and the existing--nvidia-*-rootfamily:In-tree precedent for the same read: go-nvlib's
WithPCIDevicesRoot(pkg/nvpci/nvpci.go:187,317, mock atmock.go:104) and k8sdynamic-resource-allocation'sWithFSFromRoot(which #264 builds on). Default unchanged means zero impact on real deployments.Alternatives
nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returnsNVML_ERROR_NOT_SUPPORTEDfor it, and on real hardware it signals "no NUMA" via return code (not the sysfs-1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway./sys/bus/pci/devices: possible (Add GOLANG_ARCH arg to Ubuntu 20.04 Dockerfile #264's fallback) but shadows the real PCI dir per-pod; a flag is cleaner.Scoped to
GetNumaNode(NVML full GPU + MIG, which delegates to the parent). A second hardcoded root atinternal/vgpu/pciutil.go:42(GFD vGPU detection) could adopt the same flag later. Happy to send a PR.