Merge branch 'main' into fix/9896-tui-status-config

Author: prekshivyas

.github/workflows/managed-images.yaml

@@ -43,6 +43,7 @@ on:
       - "scripts/**"
       - "src/lib/actions/sandbox/openshell-child-visible-credentials.v*.json"
       - "src/lib/actions/sandbox/mcp-bridge-*.ts"
+      - "src/lib/actions/sandbox/rebuild-post-restore-phase.ts"
       - "src/lib/core/json-types.ts"
       - "src/lib/core/ports.ts"
       - "src/lib/messaging/**"

docs/inference/set-up-vllm.mdx

@@ -570,6 +570,7 @@ The host checks the token before `docker pull`, so a missing or empty token stop
 
 Set `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` to a JSON array of individual non-blank `vllm serve` tokens for operator-owned options and overrides.
 NemoClaw trims and validates the array before downloads, shell-quotes each token, and appends the tokens after the registry defaults.
+When the array overrides `--gpu-memory-utilization`, NemoClaw validates the last value and uses that effective value for GPU-memory preflight, matching the final `vllm serve` command.
 
 ```bash
 NEMOCLAW_PROVIDER=install-vllm \
@@ -579,6 +580,10 @@ NEMOCLAW_PROVIDER=install-vllm \
 
 If the selected image does not support an argument, the managed container exits and NemoClaw prints the log tail.
 
+When the selected managed recipe defines a GPU-memory utilization value, NemoClaw requires valid `nvidia-smi` total/free-memory telemetry for the Docker-selected GPU before large downloads.
+It repeats the same check with fresh telemetry immediately before launch so another workload cannot consume the required free GPU memory unnoticed during the download.
+If telemetry is unavailable, the Docker-selected device cannot be resolved, or free memory is below the effective utilization requirement, onboarding stops with a diagnostic and can be resumed after the GPU state is corrected.
+
 To bound resource use while investigating long-context workflows on one DGX Spark or N1x host, select the Qwen profile.
 The following override disables async scheduling and lowers the context window, concurrent-sequence limit, and batch limit.
 

docs/reference/commands.mdx

@@ -1054,7 +1054,10 @@ Jetson/Tegra hosts that ship without `nvidia-smi` continue to be detected via th
 Use `--no-gpu` to opt out when you want host-side inference providers only and do not need direct GPU access inside the sandbox.
 Use `--gpu` to require GPU passthrough and fail fast if an NVIDIA GPU is not detected.
 Use `--sandbox-gpu` or `--no-sandbox-gpu` to control only direct NVIDIA GPU access inside the sandbox.
-Use `--sandbox-gpu --sandbox-gpu-device <device>` to pass a specific OpenShell GPU device selector to `openshell sandbox create`; device selectors require explicit sandbox GPU enablement.
+Use `--sandbox-gpu --sandbox-gpu-device <device>` to select an NVIDIA GPU by index (`0`), GPU UUID (`GPU-...`), or full CDI device name (`nvidia.com/gpu=0`).
+NemoClaw preserves the selection on resume.
+For native Docker and Podman creation, NemoClaw passes the normalized CDI name through OpenShell driver config; compatibility routes use the equivalent container-runtime selector.
+Device selection requires explicit sandbox GPU enablement.
 On ordinary native Linux Docker-driver hosts, NemoClaw uses native OpenShell GPU injection by default and never broadens confinement automatically.
 
 Portable onboarding requires native OpenShell GPU injection for every agent.
@@ -4893,7 +4896,7 @@ Set them before running `$$nemoclaw onboard`.
 | `NEMOCLAW_VLLM_MODEL` | registry slug or Hugging Face model ID | Selects the model the managed-vLLM install path serves and remains authoritative during DGX Station installer setup. Slugs and full model IDs are case-insensitive. Recognized slugs: `qwen3.6-27b`, `qwen3.6-35b-a3b-nvfp4`, `muse-glimmer-30b`, `nemotron-3.5-lightning-30b`, `nemotron-3-nano-4b`, `deepseek-v4-flash`, `nemotron-3-ultra-550b-a55b`, `deepseek-r1-distill-70b`. The `muse-glimmer-30b` and `nemotron-3.5-lightning-30b` profiles are Experimental on DGX Spark and Linux x86_64 with a qualifying NVIDIA GPU. NemoClaw does not enable vision or DFlash speculative decoding for Muse Glimmer. Station Express selects `nemotron-3-ultra-550b-a55b`; a qualified reciprocal pair uses the distributed topology, while no qualifying pair retains the single-Station Ultra topology. Outside Station Express, unset uses the per-platform profile default. Gated models (for example, `deepseek-r1-distill-70b`) require `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN`. |
 | `NEMOCLAW_DGX_STATION_PEER` | SSH host or `user@host` | Selects one exact, already-trusted DGX Station peer for Nemotron 3 Ultra pair qualification. The peer must match the reciprocal private `/30` rail and hardware checks; an explicit peer failure stops setup instead of falling back. NemoClaw does not enroll SSH trust or accept a port or SSH option in this value. When unset, DGX Station installer discovery checks only the two deterministic `/30` counterpart addresses. A peer cannot be combined with an explicit non-Ultra model; conflicting explicit selections fail before pair preparation. |
 | `NEMOCLAW_DGX_STATION_SSH_BINDING` | opaque installer-managed token | Carries the qualified peer endpoint and host-key binding from DGX Station pair preparation into the current managed-vLLM install. The installer creates and clears this token; operators should not set or persist it. Missing, changed, or mismatched binding state fails before peer SSH or Docker work. |
-| `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` | JSON array of non-blank strings | Appends advanced operator-owned tokens to the managed `vllm serve` command after NemoClaw's registry defaults. Example: `["--max-num-seqs","2"]`. Malformed JSON, non-string tokens, or blank tokens fail before Docker work starts. |
+| `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` | JSON array of non-blank strings | Appends advanced operator-owned tokens to the managed `vllm serve` command after NemoClaw's registry defaults. Example: `["--max-num-seqs","2"]`. Malformed JSON, non-string tokens, blank tokens, or an invalid `--gpu-memory-utilization` override fail before Docker work starts. The last memory-utilization override also controls the early and immediate pre-launch GPU-memory checks. |
 <AgentOnly variant="openclaw">
 | `NEMOCLAW_MINIMAL_BOOTSTRAP` | `1` to enable | Skips default OpenClaw workspace-template seeding for new pristine workspaces. Existing files are not deleted; refer to [Understand Runtime Changes](../manage-sandboxes/configure-sandboxes/understand-runtime-changes). |
 </AgentOnly>
@@ -5071,7 +5074,7 @@ Set them before running `$$nemoclaw onboard`.
 | `NEMOCLAW_CPU` | percentage or Kubernetes CPU quantity | Overrides the selected profile's CPU size passed to OpenShell `--cpu`. Percentages resolve against detected capacity. |
 | `NEMOCLAW_RAM` | percentage or Kubernetes memory quantity | Overrides the selected profile's memory size passed to OpenShell `--memory`. Percentages resolve against detected capacity. |
 | `NEMOCLAW_SANDBOX_GPU` | `auto`, `1`, or `0` | Controls sandbox GPU passthrough during onboarding. `auto` enables GPU passthrough when an NVIDIA GPU is detected, `1` requires GPU passthrough, and `0` forces CPU-only sandbox creation. |
-| `NEMOCLAW_SANDBOX_GPU_DEVICE` | OpenShell GPU device selector | Selects the GPU device passed with `openshell sandbox create --gpu-device`. Requires explicit sandbox GPU enablement with `NEMOCLAW_SANDBOX_GPU=1` (or `--sandbox-gpu` for CLI-driven onboarding); otherwise onboarding rejects the selector instead of treating it as an implicit opt-in. |
+| `NEMOCLAW_SANDBOX_GPU_DEVICE` | NVIDIA GPU index, UUID, or CDI device name | Selects the GPU through OpenShell driver config on native Docker and Podman routes, or through the equivalent container-runtime selector on a compatibility route. Requires explicit sandbox GPU enablement with `NEMOCLAW_SANDBOX_GPU=1` (or `--sandbox-gpu` for CLI-driven onboarding); otherwise onboarding rejects the selector instead of treating it as an implicit opt-in. |
 | `NEMOCLAW_SANDBOX_BASE_IMAGE_REFRESH` | `1`, `true`, `yes`, or `on` to enable | Bypasses recorded sandbox base-image resolution metadata during onboarding, recreation, and rebuild. NemoClaw reruns candidate resolution but can still use a compatible image from Docker's local image store. Versioned release candidates that exist locally but fail validation are refreshed from the registry once during normal resolution. This setting does not discard onboarding session state. |
 | `NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD` | unset or `auto` (default); `1`, `true`, `yes`, or `on` to enable; `0`, `false`, `no`, or `off` to disable | Controls whether base-image resolution may build a compatible image locally. The default allows builds during normal CLI runs and disables them when `NODE_ENV=test` or `VITEST=true`. When source inputs or a missing/incompatible release-version base require a fresh build, disabling local builds makes resolution fail instead of using an unproven image. |
 | `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `fallback`, `1`, or `0`; other legacy nonzero values remain accepted through `v0.0.x` and will be removed in `v0.1.0` | Selects Linux Docker-driver GPU routing. Unset, `auto`, or `0` uses native OpenShell GPU injection on ordinary native Linux. `fallback` explicitly opts into one native attempt followed by one bounded compatibility retry when trusted host evidence identifies a GPU-routing failure. `1` and legacy nonzero values select the compatibility patch from the outset. Docker Desktop WSL and Jetson/Tegra use the compatibility path by default; Docker Desktop WSL ignores `0`, while Jetson/Tegra accepts `0` only as a troubleshooting override that bypasses device-group propagation. Portable onboarding requires native OpenShell GPU injection for every agent and does not use this setting. |

src/lib/actions/sandbox/destroy-flow.test.ts

@@ -446,6 +446,40 @@ describe("destroySandbox flow", () => {
     );
   });
 
+  it("releases lifecycle locks before exiting on Portable identity drift and permits retry", async () => {
+    const harness = createDestroyHarness({ portableDestroyAuthority: true });
+    const recordRevalidation = () => harness.events.push("portable-revalidate");
+    const revalidateAtDeleteBoundary = vi.fn(recordRevalidation).mockImplementationOnce(() => {
+      throw new Error("Portable receipt changed during destroy");
+    });
+    harness.portableDestroyRevalidateSpy.mockImplementation(() =>
+      harness.events.at(-1) === "detach" ? revalidateAtDeleteBoundary() : recordRevalidation(),
+    );
+    exitSpy.mockImplementationOnce(((code?: number | string | null) => {
+      harness.lifecycleLockEvents.push("process-exit");
+      throw new Error(`process.exit(${code ?? 0})`);
+    }) as never);
+
+    await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow("process.exit(1)");
+
+    expect(harness.lifecycleLockEvents).toEqual(["acquired", "released", "process-exit"]);
+    expect(harness.events).not.toContain("delete");
+    expect(harness.removeSandboxSpy).not.toHaveBeenCalled();
+    expect(harness.retirePortableLifecycleReceiptSpy).not.toHaveBeenCalled();
+
+    await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();
+    expect(harness.lifecycleLockEvents).toEqual([
+      "acquired",
+      "released",
+      "process-exit",
+      "acquired",
+      "released",
+    ]);
+    expect(harness.events.filter((event) => event === "delete")).toHaveLength(1);
+    expect(harness.removeSandboxSpy).toHaveBeenCalledOnce();
+    expect(harness.retirePortableLifecycleReceiptSpy).toHaveBeenCalledOnce();
+  });
+
   it("redacts credentials from an invalid schema-4 Portable receipt refusal and releases the lifecycle lock before retry (#9189)", async () => {
     const harness = createDestroyHarness({
       portableDestroyPrepareError: "portable receipt API_KEY=opaque-portable-secret is invalid",

src/lib/actions/sandbox/destroy.ts

@@ -464,14 +464,30 @@ export type { WipeSandboxStateDeps };
 // the wipe was extracted out of the destroy monolith (#5455 PRA-2).
 export { wipeSandboxState };
 
+class SandboxDestroyExitRequest extends Error {
+  constructor(readonly exitCode: number) {
+    super(`Sandbox destroy requested exit ${String(exitCode)}`);
+    this.name = "SandboxDestroyExitRequest";
+  }
+}
+
+function requestSandboxDestroyExit(exitCode: number): never {
+  throw new SandboxDestroyExitRequest(exitCode);
+}
+
 export async function destroySandbox(
   sandboxName: string,
   options: string[] | DestroySandboxOptions = {},
 ): Promise<void> {
-  return withMcpLifecycleLock(sandboxName, () => {
-    assertSandboxDestroyCommandAvailable(sandboxName);
-    return destroySandboxUnlocked(sandboxName, options);
-  });
+  try {
+    return await withMcpLifecycleLock(sandboxName, () => {
+      assertSandboxDestroyCommandAvailable(sandboxName);
+      return destroySandboxUnlocked(sandboxName, options);
+    });
+  } catch (error) {
+    if (error instanceof SandboxDestroyExitRequest) process.exit(error.exitCode);
+    throw error;
+  }
 }
 
 async function destroySandboxUnlocked(
@@ -509,7 +525,7 @@ async function destroySandboxUnlocked(
     });
   const initialIdentity = portableContainerAuthority ? null : inspectContainerIdentity();
   if (initialIdentity === false) {
-    process.exit(1);
+    requestSandboxDestroyExit(1);
   }
 
   const registeredSandbox = registry.getSandbox(sandboxName);
@@ -553,7 +569,7 @@ async function destroySandboxUnlocked(
       console.error(
         `  The sandbox was not deleted. Resolve the reported cleanup preflight failure and retry destroy.`,
       );
-      process.exit(1);
+      requestSandboxDestroyExit(1);
     }
   }
   const abortPreparedCleanupOnError = <T>(operation: () => T): T => {
@@ -576,7 +592,7 @@ async function destroySandboxUnlocked(
       console.error(
         `  Refusing to destroy sandbox '${sandboxName}': NemoClaw could not revalidate Portable container identity during preflight: ${redactDestroyError(error)}. NemoClaw removed no sandbox resources.`,
       );
-      process.exit(1);
+      requestSandboxDestroyExit(1);
     }
   } else {
     const preMutationIdentity = abortPreparedCleanupOnError(inspectContainerIdentity);
@@ -590,7 +606,7 @@ async function destroySandboxUnlocked(
         );
       }
       preparedManagedLlamaCppCleanup?.abort();
-      process.exit(1);
+      requestSandboxDestroyExit(1);
     }
   }
   const priorHttpsPinRouteId = abortPreparedCleanupOnError(() =>
@@ -675,7 +691,7 @@ async function destroySandboxUnlocked(
       }
     }
     preparedManagedLlamaCppCleanup?.abort();
-    process.exit(destructiveResult.exitCode);
+    requestSandboxDestroyExit(destructiveResult.exitCode);
   }
   const {
     detachOutcome,
@@ -738,7 +754,7 @@ async function destroySandboxUnlocked(
       );
       console.error("  The sandbox registry entry was preserved so exact cleanup can be retried.");
       preparedManagedLlamaCppCleanup?.abort();
-      process.exit(1);
+      requestSandboxDestroyExit(1);
     }
     if (stateVolumeCleanup.status === "not-owned") {
       console.warn(
@@ -770,7 +786,7 @@ async function destroySandboxUnlocked(
         `  Managed llama.cpp cleanup failed for '${sandboxName}': ${managedLlamaCppCleanup.reason}`,
       );
       console.error("  The sandbox registry entry was preserved so exact cleanup can be retried.");
-      process.exit(1);
+      requestSandboxDestroyExit(1);
     }
   }
   // The sandbox's gateway was captured before the registry entry is removed —
@@ -814,7 +830,7 @@ async function destroySandboxUnlocked(
     emitProviderDetachResidualHint(sandboxName, detachOutcome.failures, (message) =>
       console.warn(`  ${YW}⚠${R}${message}`),
     );
-    process.exit(1);
+    requestSandboxDestroyExit(1);
   }
   if (removed) {
     try {