Skip to content

Commit ef934fb

Browse files
authored
Deduplicate the wedge remediation docs
The drain section restated the quarantine's placement-exclusion semantics and the vf-health.json edit-plus-restart rule, and gave the uncordon instruction twice. Hoist the edit rule into one paragraph both the cordon and clear steps reference, and let the detection paragraph own the exclusion semantics.
1 parent af322ba commit ef934fb

1 file changed

Lines changed: 19 additions & 28 deletions

File tree

lib/devices/GPU.md

Lines changed: 19 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -286,12 +286,11 @@ NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x22:0x65:884)
286286

287287
Hypeman detects this automatically: the guest agent watches the guest kernel
288288
log (`/dev/kmsg`) for that line and reports it as a `HYPEMAN-GPU-INIT-FAILED`
289-
marker — the same guest-to-host channel as the other `HYPEMAN-*` markers,
290-
landing in the instance's `logs/app.log` — and the vGPU sentinel controller
291-
scans that file for every vendor VFIO instance. A match quarantines the VF in
289+
marker in the instance's `logs/app.log`, which the vGPU sentinel controller
290+
scans for every vendor VFIO instance. A match quarantines the VF in
292291
`<data-dir>/gpu/vf-health.json` (it survives restarts): the VF is excluded
293292
from placement and from advertised profile availability, and its parent GPU
294-
becomes overflow-only so it drains toward the SR-IOV cycle. The conviction is
293+
becomes overflow-only — deprioritized for new placements. The conviction is
295294
logged at error level (`quarantined wedged vGPU VF`) and counted in
296295
`hypeman_instances_vgpu_sentinel_convictions_total`;
297296
`hypeman_instances_vgpu_quarantined_vfs` gauges the current quarantine count.
@@ -330,26 +329,20 @@ requires no vGPU assignments on that GPU). The DCGM quiesce is not optional:
330329
with `nv-hostengine`/`dcgm-exporter` holding the GPUs open, `sriov-manage -d`
331330
fails with `Cannot obtain unbindLock` on first contact.
332331

333-
**Draining the parent GPU.** The quarantine only deprioritizes the card
334-
(overflow-only), so under capacity pressure new placements can still land on
335-
its healthy VFs and refill it. To actually drain the card, cordon it by
336-
adding every one of its VFs to `<data-dir>/gpu/vf-health.json` (copy the
337-
record shape of a real conviction: `vf_address` plus `quarantined_at`) and
338-
restarting hypeman immediately — the store loads only at startup, and a
339-
conviction landing before the restart re-persists the in-memory set over
340-
your edit. The restart does not disturb running VMs: startup reconciliation
341-
protects live VFs. Quarantined VFs are fully excluded from placement and
342-
from advertised profile availability, so the cordon holds even when every
343-
other card is full, and `hypeman_instances_vgpu_quarantined_vfs` reads high
344-
for its duration — expected, not an incident.
345-
346-
Running instances are untouched by the cordon and drain through their normal
347-
lifecycle: standby is blocked for vGPU instances, so only a running VM pins
348-
a VF, every stop or delete releases it (never to be re-selected while
349-
cordoned), and the next start picks a fresh VF on another card. Monitor the
350-
drain by listing instances whose `gpu.device_path` sits under the parent
351-
GPU; once none remain, run the cycle below, then remove the card's entries
352-
from `vf-health.json` and restart hypeman again to uncordon.
332+
Any manual edit to `vf-health.json` needs an immediate hypeman restart: the
333+
store loads only at startup, and a conviction landing first re-persists the
334+
in-memory set over your edit. The restart does not disturb running VMs —
335+
startup reconciliation protects live VFs.
336+
337+
**Draining the parent GPU.** Overflow-only is a preference, not a cordon:
338+
under capacity pressure new placements still land on the card's healthy VFs
339+
and refill it. To drain the card, quarantine all of its VFs by hand — add
340+
records to `vf-health.json` (the shape of a real conviction: `vf_address`
341+
plus `quarantined_at`) and restart. Running instances are untouched and
342+
drain through their normal lifecycle: standby is blocked for vGPU instances,
343+
so only a running VM pins a VF, and each stop or delete frees one for good.
344+
Monitor by listing instances whose `gpu.device_path` sits under the parent
345+
GPU; once none remain, run the cycle below.
353346

354347
```bash
355348
# 1. Quiesce the services holding the GPU (required for the unbind lock).
@@ -363,10 +356,8 @@ systemctl stop nvidia-dcgm-exporter nvidia-dcgm
363356
systemctl start nvidia-dcgm nvidia-dcgm-exporter
364357
```
365358

366-
After the cycle, clear the quarantine by removing the VF's entry from
367-
`<data-dir>/gpu/vf-health.json` and restarting hypeman immediately — the
368-
running process keeps the quarantine in memory, and a conviction landing
369-
before the restart re-persists it over your edit. Then boot a GPU instance as
359+
After the cycle, clear the quarantine by removing the card's entries from
360+
`vf-health.json` (restart rule above), then boot a GPU instance as
370361
verification: placement excludes quarantined VFs, so the recovered VF cannot
371362
be targeted while its entry exists, and there is no VF-pin API — clearing
372363
first is safe because the sentinel automatically re-quarantines the VF if the

0 commit comments

Comments
 (0)