feat: reconnect-ready QEMU transports and VM.attach() for host crash recovery

[ShravanSunder]

Problem

When the host process dies unexpectedly, QEMU keeps running (reparented to PID 1), but Gondolin has no way to re-own that live VM from a new host process. The guest-side sandboxd daemon survives and waits for reconnection, and the host-side VirtioBridge can recreate its listeners — but transport socket paths are random UUIDs that die with the process, and there is no metadata to discover the orphaned QEMU. Downstream systems are forced to treat any host crash as full VM loss, even when the guest is still alive.

#88

Solution

Add deterministic transport identity and a VM.attach() API so a new host process can recover ownership of an already-running VM.

How it works

1. Deterministic runtime directory

All QEMU transport sockets are now placed under a stable per-VM directory:

/tmp/gondolin-runtime/{vm.id}/
  ├── virtio.sock
  ├── virtio-fs.sock
  ├── virtio-ssh.sock
  ├── virtio-ingress.sock
  ├── net.sock
  └── runtime.json

Previously each socket used a random UUID suffix (gondolin-virtio-{random}.sock). Now they derive from the VM's stable id, so a new process can reconstruct the exact same paths.

2. Reconnect-ready QEMU sockets

When QEMU >= 9.2 is detected (via --version parsing), chardev and netdev socket args include reconnect-ms=5000. This tells QEMU to retry the socket connection every 5 seconds after a disconnect, instead of giving up permanently. Detection is automatic and falls back gracefully on older QEMU versions.

3. Runtime metadata

After VM start, a runtime.json file is written to the runtime directory containing the VM id, QEMU PID, binary path, creation timestamp, and reconnect capability. This is the discovery mechanism for VM.attach(). The file is removed on VM.close().

4. VM.attach({ id, ...options })

New static method that:

1. Reads and validates runtime metadata for the given VM id
2. Verifies the QEMU process is still alive (ESRCH = dead, EPERM = alive but different user)
3. Rejects VMs created without reconnect support
4. Resolves sandbox options with the same deterministic runtime ID (same socket paths)
5. Creates a SandboxServer in attach mode — SandboxController skips spawn, transitions directly to "running"
6. Host-side VirtioBridge recreates its Unix socket listeners at the original paths
7. QEMU reconnects on its next retry cycle, sandboxd exits its poll-wait, full communication restored

5. Attach-mode process lifecycle

SandboxController with an attachedPid:

• start() verifies the PID is alive before reporting "running"
• close() sends SIGTERM, polls for exit, escalates to SIGKILL after 3s, hard timeout at 10s
• Signal errors are handled with proper ESRCH/EPERM discrimination (shared isProcessAlive/signalProcess utilities)

6. Guest-side non-blocking open

sandboxd's tryOpenRpcPath() now opens virtio ports with O_NONBLOCK (then clears it via fcntl), with errdefer to prevent fd leaks. This avoids blocking indefinitely on a dead virtio port during the reconnect window.

What VM.attach() does NOT do

• Recover in-flight exec/file operations from the crashed process — those are lost
• Change the existing gondolin attach CLI session-IPC behavior
• Claim support for QEMU versions that lack reconnect-ms (< 9.2)

Type design

• VMAttachOptions uses Pick<VMOptions, ...> to expose only attach-relevant fields (excludes memory, cpus, autoStart, rootfs which are fixed by the running VM)
• VMConstructionOptions is a discriminated union (mode: "create" | mode: "attach") preventing invalid field combinations
• VmRuntimeMetadata validates all fields including qemuPid > 0 and wraps JSON.parse with contextual error messages

Testing

Automated unit tests (pnpm --dir host test)

These run without hardware virtualization and are part of the standard test suite:

test/qemu-arch-mismatch.test.ts — option resolution

• Deterministic socket paths derived from runtime ID
• parseQemuVersion — valid versions (9.2, 10.2), malformed input

test/sandbox-controller.test.ts — attach-mode controller behavior

• buildQemuArgs — reconnect-ms appended to all chardevs and netdev
• Attach mode: no spawn, state transitions to running
• Attach mode: PID liveness check before reporting running
• Attach mode: EPERM treated as alive (process exists, different user)
• Attach mode: stale PID rejected before reporting running
• Attach mode: close with SIGTERM → SIGKILL escalation

test/vm-internals.test.ts — VM.attach and metadata lifecycle

• Constructor derives runtime identity and socket paths from VM id
• VM.attach rejects missing runtime metadata
• VM.attach rejects corrupt runtime metadata
• VM.attach rejects non-reconnect-capable runtimes
• VM.attach rejects stale PIDs (ESRCH)
• VM.attach reuses runtime metadata id and attached QEMU pid
• Runtime metadata written on start, removed on close

Live reconnect validation (pnpm --dir host test:attach-reconnect)

This is a separate validation script that requires real QEMU >= 9.2 with hardware virtualization. It is not part of the standard pnpm test suite — it must be run manually on a machine that can boot VMs. The script (scripts/validate-vm-attach-reconnect.ts) with its helper (test/helpers/vm-attach-helper.ts) exercises a real crash/reattach flow:

1. Helper process creates a VM, writes a marker file, starts a long-lived background process, verifies mapped TCP egress works
2. Helper prints the vm.id to stdout, then keeps the VM alive
3. Validator kills the helper with SIGKILL (simulating a host crash) — QEMU stays alive (reparented to PID 1)
4. Validator calls VM.attach({ id }) from a new process
5. After attach, the validator confirms:
   • guest marker file (/tmp/reconnect-marker) still exists
   • long-lived guest background process is still alive (PID check via kill -0)
   • fresh vm.exec(...) works
   • fresh mapped TCP egress works (new HTTP request through host-mapped port)
   • vm.close() shuts the orphaned QEMU down cleanly

Latest reconnect validation output:

helper vm id: 31c9aa97-e506-4da9-ad35-016b55d23728
reconnect validation passed

Build and existing test suite

All validation steps were run sequentially:

1. make build → exit 0 ✅
2. make check → exit 0 ✅
3. make test → exit 0 ✅
4. pnpm --dir host test — 464 pass, 4 fail (pre-existing: require hardware virtualization), 0 regressions
5. pnpm --dir host test:attach-reconnect → exit 0 ✅

Closes #88

🤖 Generated with Claude Code