@@ -286,12 +286,11 @@ NVRM: GPU 0000:00:03.0: RmInitAdapter failed! (0x22:0x65:884)
286286
287287Hypeman detects this automatically: the guest agent watches the guest kernel
288288log (` /dev/kmsg ` ) for that line and reports it as a ` HYPEMAN-GPU-INIT-FAILED `
289- marker — the same guest-to-host channel as the other ` HYPEMAN-* ` markers,
290- landing in the instance's ` logs/app.log ` — and the vGPU sentinel controller
291- scans that file for every vendor VFIO instance. A match quarantines the VF in
289+ marker in the instance's ` logs/app.log ` , which the vGPU sentinel controller
290+ scans for every vendor VFIO instance. A match quarantines the VF in
292291` <data-dir>/gpu/vf-health.json ` (it survives restarts): the VF is excluded
293292from placement and from advertised profile availability, and its parent GPU
294- becomes overflow-only so it drains toward the SR-IOV cycle . The conviction is
293+ becomes overflow-only — deprioritized for new placements . The conviction is
295294logged at error level (` quarantined wedged vGPU VF ` ) and counted in
296295` hypeman_instances_vgpu_sentinel_convictions_total ` ;
297296` hypeman_instances_vgpu_quarantined_vfs ` gauges the current quarantine count.
@@ -330,26 +329,20 @@ requires no vGPU assignments on that GPU). The DCGM quiesce is not optional:
330329with ` nv-hostengine ` /` dcgm-exporter ` holding the GPUs open, ` sriov-manage -d `
331330fails with ` Cannot obtain unbindLock ` on first contact.
332331
333- ** Draining the parent GPU.** The quarantine only deprioritizes the card
334- (overflow-only), so under capacity pressure new placements can still land on
335- its healthy VFs and refill it. To actually drain the card, cordon it by
336- adding every one of its VFs to ` <data-dir>/gpu/vf-health.json ` (copy the
337- record shape of a real conviction: ` vf_address ` plus ` quarantined_at ` ) and
338- restarting hypeman immediately — the store loads only at startup, and a
339- conviction landing before the restart re-persists the in-memory set over
340- your edit. The restart does not disturb running VMs: startup reconciliation
341- protects live VFs. Quarantined VFs are fully excluded from placement and
342- from advertised profile availability, so the cordon holds even when every
343- other card is full, and ` hypeman_instances_vgpu_quarantined_vfs ` reads high
344- for its duration — expected, not an incident.
345-
346- Running instances are untouched by the cordon and drain through their normal
347- lifecycle: standby is blocked for vGPU instances, so only a running VM pins
348- a VF, every stop or delete releases it (never to be re-selected while
349- cordoned), and the next start picks a fresh VF on another card. Monitor the
350- drain by listing instances whose ` gpu.device_path ` sits under the parent
351- GPU; once none remain, run the cycle below, then remove the card's entries
352- from ` vf-health.json ` and restart hypeman again to uncordon.
332+ Any manual edit to ` vf-health.json ` needs an immediate hypeman restart: the
333+ store loads only at startup, and a conviction landing first re-persists the
334+ in-memory set over your edit. The restart does not disturb running VMs —
335+ startup reconciliation protects live VFs.
336+
337+ ** Draining the parent GPU.** Overflow-only is a preference, not a cordon:
338+ under capacity pressure new placements still land on the card's healthy VFs
339+ and refill it. To drain the card, quarantine all of its VFs by hand — add
340+ records to ` vf-health.json ` (the shape of a real conviction: ` vf_address `
341+ plus ` quarantined_at ` ) and restart. Running instances are untouched and
342+ drain through their normal lifecycle: standby is blocked for vGPU instances,
343+ so only a running VM pins a VF, and each stop or delete frees one for good.
344+ Monitor by listing instances whose ` gpu.device_path ` sits under the parent
345+ GPU; once none remain, run the cycle below.
353346
354347``` bash
355348# 1. Quiesce the services holding the GPU (required for the unbind lock).
@@ -363,10 +356,8 @@ systemctl stop nvidia-dcgm-exporter nvidia-dcgm
363356systemctl start nvidia-dcgm nvidia-dcgm-exporter
364357```
365358
366- After the cycle, clear the quarantine by removing the VF's entry from
367- ` <data-dir>/gpu/vf-health.json ` and restarting hypeman immediately — the
368- running process keeps the quarantine in memory, and a conviction landing
369- before the restart re-persists it over your edit. Then boot a GPU instance as
359+ After the cycle, clear the quarantine by removing the card's entries from
360+ ` vf-health.json ` (restart rule above), then boot a GPU instance as
370361verification: placement excludes quarantined VFs, so the recovered VF cannot
371362be targeted while its entry exists, and there is no VF-pin API — clearing
372363first is safe because the sentinel automatically re-quarantines the VF if the
0 commit comments