Skip to content

[Feature]: Add a configurable --sysfs-root for device NUMA detection (GetNumaNode) #1833

Description

@eliranw

Summary

GetNumaNode reads a device's NUMA node from a hardcoded /sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.
This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same --sysfs-root request for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).

Problem

internal/rm/nvml_devices.go:150-162:

busID := strings.ToLower(strings.TrimPrefix(uint8Slice(info.BusId[:]).String(), "0000"))
b, err := os.ReadFile(fmt.Sprintf("/sys/bus/pci/devices/%s/numa_node", busID))
if err != nil {
    return false, 0, nil  // error swallowed -> device reported with no NUMA
}

The /sys prefix is a literal; nothing overrides it (--nvidia-driver-root / --nvidia-dev-root feed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails, hasNuma=false, and BuildDevice (internal/rm/devices.go:93-101) leaves Device.Topology nil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.

Proposed solution

Flag --sysfs-root (default /sys), matching #264 and the existing --nvidia-*-root family:

os.ReadFile(filepath.Join(sysfsRoot, "bus/pci/devices", busID, "numa_node"))

In-tree precedent for the same read: go-nvlib's WithPCIDevicesRoot (pkg/nvpci/nvpci.go:187,317, mock at mock.go:104) and k8s dynamic-resource-allocation's WithFSFromRoot (which #264 builds on). Default unchanged means zero impact on real deployments.

Alternatives

  • nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returns NVML_ERROR_NOT_SUPPORTED for it, and on real hardware it signals "no NUMA" via return code (not the sysfs -1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway.
  • Subpath-mount the tree over /sys/bus/pci/devices: possible (Add GOLANG_ARCH arg to Ubuntu 20.04 Dockerfile #264's fallback) but shadows the real PCI dir per-pod; a flag is cleaner.

Scoped to GetNumaNode (NVML full GPU + MIG, which delegates to the parent). A second hardcoded root at internal/vgpu/pciutil.go:42 (GFD vGPU detection) could adopt the same flag later. Happy to send a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions