VM-tool-shuttle-workflow-codex-test-runbook.txt

VM TOOL SHUTTLE WORKFLOW - CODEX TEST RUNBOOK
=============================================

Purpose
-------
Use this runbook when asking Codex to create a small test VM on the local
kg55 Debian/KVM system, prepare it for portability, convert it from qcow2 to
VMDK, and produce an evidence bundle for transfer/import testing on a remote
VMware ESXi system.

This is a lab workflow. Do not touch production VMs. Do not overwrite existing
VM disks. Stop and report if any required host, network, storage, or credential
detail is missing.


Expected Operator Prompt
------------------------
Paste or reference this runbook, then ask:

  Create a small test VM using this VM Tool Shuttle workflow. Build it on kg55,
  prepare it for transfer to my remote ESXi system, convert the powered-off VM
  disk to VMDK, and leave me an evidence bundle with commands, outputs,
  checksums, and next-step import instructions. Stop before contacting ESXi
  unless I provide the ESXi host and credentials.


Default Test VM Values
----------------------
VM_NAME: vm-shuttle-test01
GUEST_OS: Debian stable minimal server
ARCH: amd64 / x86_64
FIRMWARE: UEFI / OVMF
MACHINE_TYPE: q35
VCPU: 2
RAM_MB: 4096
DISK_GB: 20
DISK_FORMAT_SOURCE: qcow2
DISK_FORMAT_EXPORT: vmdk
DISK_BUS_ON_KVM: virtio
NETWORK_ON_KVM: virtio
NETWORK_MODE: bridge br0 preferred; NAT acceptable for first local toolchain test
PARTITION_TABLE: GPT
FILESYSTEM: ext4
SOURCE_HOST_LABEL: kg55
ESXI_IMPORT_MODE: manual first, automation later


Required Inputs To Confirm Before Acting
----------------------------------------
Codex must collect or infer these values before creating anything:

1. Debian/KVM host access path.
   Example: local shell on kg55, SSH to kg55, or an already-open terminal.

2. ISO or cloud image source.
   Example: /srv/isos/debian-*.iso or a Debian cloud image URL/path.

3. VM storage location.
   Example: /var/lib/libvirt/images or /srv/vms.

4. Network choice.
   Preferred: bridge br0.
   Fallback: libvirt default NAT network.

5. Transfer target policy.
   For this test, produce local artifacts first. Do not contact ESXi unless
   the operator explicitly supplies ESXi host, datastore, and credentials.


Hard Stop Conditions
--------------------
Stop and report before making changes if any of these are true:

- CPU virtualization support is absent.
- libvirt/KVM commands are unavailable.
- OVMF/UEFI firmware is unavailable.
- Disk free space is less than DISK_GB plus 2x export working room.
- The chosen VM_NAME already exists and the operator has not approved reuse.
- The target VM disk path already exists and the operator has not approved reuse.
- No ISO/cloud image source is available.
- The selected network bridge does not exist and NAT fallback is not acceptable.
- Any command would modify an unrelated existing VM.
- ESXi contact, upload, or import is requested but ESXi credentials are missing.


Phase 0 - Create Working Directories
------------------------------------
Recommended layout:

  /srv/vms/vm-shuttle-test01/
    vm-shuttle-test01.qcow2
    vm-shuttle-test01-libvirt.xml
    logs/
    exports/
      YYYYMMDD-HHMM/
        vm-shuttle-test01.qcow2
        vm-shuttle-test01.vmdk
        qcow2-info.txt
        vmdk-info.txt
        SHA256SUMS
        commands.log
        notes.md

Commands, adjust BASE_DIR if needed:

  VM_NAME=vm-shuttle-test01
  BASE_DIR=/srv/vms/${VM_NAME}
  RUN_ID=$(date +%Y%m%d-%H%M)
  EXPORT_DIR=${BASE_DIR}/exports/${RUN_ID}
  sudo mkdir -p "${BASE_DIR}/logs" "${EXPORT_DIR}"
  sudo chown -R "$USER":"$USER" "${BASE_DIR}"


Phase 1 - Host Preflight
------------------------
Run:

  hostnamectl
  uname -a
  egrep -c '(vmx|svm)' /proc/cpuinfo
  command -v virsh
  command -v virt-install
  command -v qemu-img
  systemctl status libvirtd --no-pager
  virsh list --all
  virsh net-list --all
  ls /usr/share/OVMF /usr/share/ovmf 2>/dev/null || true
  df -h /srv /var/lib/libvirt/images "$HOME" 2>/dev/null

Pass criteria:

- virtualization count is 1 or greater
- virsh, virt-install, and qemu-img exist
- libvirt is active or socket-activated
- OVMF firmware files exist
- enough disk space exists for qcow2 plus exported copy plus VMDK

If packages are missing and operator approves installation:

  sudo apt update
  sudo apt install -y qemu-kvm libvirt-daemon-system virt-manager ovmf bridge-utils qemu-utils libguestfs-tools virtinst


Phase 2 - Safety Checks
-----------------------
Run:

  virsh dominfo "${VM_NAME}" >/tmp/${VM_NAME}.dominfo 2>&1; echo $?
  test -e "${BASE_DIR}/${VM_NAME}.qcow2"; echo $?

Pass criteria:

- virsh dominfo should fail if the VM does not already exist
- disk path should not already exist

If either exists, stop and ask whether to use a different VM_NAME or remove/reuse
the existing lab VM.


Phase 3 - Create Test Disk
--------------------------
Run:

  qemu-img create -f qcow2 "${BASE_DIR}/${VM_NAME}.qcow2" 20G
  qemu-img info "${BASE_DIR}/${VM_NAME}.qcow2"

Pass criteria:

- qcow2 file exists
- virtual size is 20 GiB or chosen DISK_GB
- format is qcow2


Phase 4 - Create Or Install Test VM
-----------------------------------
Preferred hardware:

- UEFI/OVMF
- q35
- 2 vCPU
- 4096 MB RAM
- VirtIO disk
- VirtIO network
- SPICE or serial console for local visibility

Example virt-install using bridge br0 and an ISO:

  virt-install \
    --name "${VM_NAME}" \
    --memory 4096 \
    --vcpus 2 \
    --machine q35 \
    --boot uefi \
    --disk path="${BASE_DIR}/${VM_NAME}.qcow2",format=qcow2,bus=virtio \
    --network bridge=br0,model=virtio \
    --graphics spice \
    --cdrom /path/to/debian.iso

Fallback NAT network example:

  virt-install \
    --name "${VM_NAME}" \
    --memory 4096 \
    --vcpus 2 \
    --machine q35 \
    --boot uefi \
    --disk path="${BASE_DIR}/${VM_NAME}.qcow2",format=qcow2,bus=virtio \
    --network network=default,model=virtio \
    --graphics spice \
    --cdrom /path/to/debian.iso

Codex instruction:

- Do not invent the ISO path. Locate one or ask the operator.
- If using a cloud image instead of ISO install, document the exact image source
  and cloud-init seed method.


Phase 5 - Guest Preparation And SSH Access
------------------------------------------
Inside the Linux guest, install portability agents and SSH access tooling:

  sudo apt update
  sudo apt install -y qemu-guest-agent open-vm-tools cloud-init openssh-server
  sudo systemctl enable --now qemu-guest-agent
  sudo systemctl enable --now open-vm-tools || true
  sudo mkdir -p /run/sshd
  sudo install -d -m 0755 /etc/ssh/sshd_config.d
  printf 'PasswordAuthentication yes\nKbdInteractiveAuthentication yes\n' | sudo tee /etc/ssh/sshd_config.d/99-codex-password-auth.conf
  sudo sshd -t
  sudo systemctl enable --now ssh

For cloud-image based automated tests, include these items in cloud-init user-data:

  packages:
    - qemu-guest-agent
    - open-vm-tools
    - cloud-init
    - openssh-server
  runcmd:
    - mkdir -p /run/sshd
    - install -d -m 0755 /etc/ssh/sshd_config.d
    - printf 'PasswordAuthentication yes\nKbdInteractiveAuthentication yes\n' > /etc/ssh/sshd_config.d/99-codex-password-auth.conf
    - sshd -t
    - systemctl enable --now ssh
    - systemctl enable --now qemu-guest-agent
    - systemctl enable --now open-vm-tools || true

If SSH is repaired offline with virt-customize, do not write literal \n strings into sshd config. Use separate real lines or a here-doc. A bad one-line drop-in such as `PasswordAuthentication yes\nKbdInteractiveAuthentication yes` will make sshd fail to start.

Collect guest evidence:

  hostnamectl
  uname -a
  lsblk -f
  ip addr
  systemctl --no-pager status ssh qemu-guest-agent open-vm-tools
  ss -ltnp | grep ':22'

Pass criteria:

- guest boots on KVM
- qemu-guest-agent installed and active on KVM
- ssh service installed, active, and listening on TCP/22
- login works with the expected lab user before calling the test complete
- open-vm-tools installed for later ESXi boot; it may be inactive while running under KVM
- disk is GPT/ext4 unless another filesystem was intentionally selected


Phase 6 - Clean Shutdown And Freeze Point
-----------------------------------------
Shutdown:

  virsh shutdown "${VM_NAME}"
  sleep 10
  virsh domstate "${VM_NAME}"

If still running after a reasonable wait:

  virsh domstate "${VM_NAME}"

Stop and ask before using destroy. Do not force power off unless operator approves.

Export libvirt XML:

  virsh dumpxml "${VM_NAME}" > "${BASE_DIR}/${VM_NAME}-libvirt.xml"
  cp "${BASE_DIR}/${VM_NAME}-libvirt.xml" "${EXPORT_DIR}/"

Pass criteria:

- VM state is shut off
- libvirt XML exists in base directory and export directory


Phase 7 - Copy Master qcow2 To Export Directory
-----------------------------------------------
Run:

  cp --reflink=auto "${BASE_DIR}/${VM_NAME}.qcow2" "${EXPORT_DIR}/${VM_NAME}.qcow2"
  qemu-img info "${EXPORT_DIR}/${VM_NAME}.qcow2" > "${EXPORT_DIR}/qcow2-info.txt"
  sha256sum "${EXPORT_DIR}/${VM_NAME}.qcow2" > "${EXPORT_DIR}/SHA256SUMS"

Pass criteria:

- exported qcow2 exists
- qemu-img info can read it
- SHA256SUMS contains qcow2 hash


Phase 8 - Convert qcow2 To VMDK
-------------------------------
Run:

  qemu-img convert -p -f qcow2 -O vmdk "${EXPORT_DIR}/${VM_NAME}.qcow2" "${EXPORT_DIR}/${VM_NAME}.vmdk"
  qemu-img info "${EXPORT_DIR}/${VM_NAME}.vmdk" > "${EXPORT_DIR}/vmdk-info.txt"
  sha256sum "${EXPORT_DIR}/${VM_NAME}.vmdk" >> "${EXPORT_DIR}/SHA256SUMS"

If ESXi later rejects this VMDK, repeat with an ESXi-compatible VMDK subformat
appropriate to the local qemu-img version and ESXi version. Record the exact
working command in notes.md.

Pass criteria:

- VMDK exists
- qemu-img info can read it
- SHA256SUMS contains VMDK hash


Phase 9 - Create Evidence Notes
-------------------------------
Create:

  ${EXPORT_DIR}/notes.md

Required contents:

- VM_NAME
- host name and OS
- exact ISO/cloud image used
- VM disk path
- network mode used
- firmware mode
- machine type
- CPU/RAM/disk values
- qemu-img version
- qemu-img info summaries
- SHA256SUMS contents
- any deviations from this runbook
- whether ESXi was contacted
- next manual ESXi import steps

Helpful command:

  qemu-img --version >> "${EXPORT_DIR}/notes.md"
  cat "${EXPORT_DIR}/qcow2-info.txt" >> "${EXPORT_DIR}/notes.md"
  cat "${EXPORT_DIR}/vmdk-info.txt" >> "${EXPORT_DIR}/notes.md"
  cat "${EXPORT_DIR}/SHA256SUMS" >> "${EXPORT_DIR}/notes.md"


Phase 10 - Optional Transfer To ESXi
------------------------------------
Do not perform this phase unless the operator explicitly supplies:

- ESXi host/IP
- datastore name
- credentials or an approved authenticated method
- target folder/path
- whether to upload via datastore browser, scp, govc, or another tool

Manual ESXi import recommendation:

1. Upload VMDK and SHA256SUMS to a lab datastore.
2. Create a new VM named vm-shuttle-test01-esxi.
3. Choose Linux guest type matching the source distro.
4. Select EFI firmware.
5. Match or exceed CPU/RAM.
6. Attach uploaded VMDK as boot disk.
7. Start with a conservative virtual disk/network controller if needed.
8. Power on and watch the console.
9. Verify login prompt, disk visibility, network, and open-vm-tools.

Do not delete or overwrite any ESXi VM as part of this simple test.


Acceptance Criteria
-------------------
The test is successful when:

- KVM host preflight passes.
- A new lab VM boots on KVM.
- The guest has qemu-guest-agent and open-vm-tools installed.
- The guest receives a LAN-visible IPv4 address when bridge br0 is selected.
- The guest runs openssh-server, listens on TCP/22, and accepts login for the expected lab user.
- IP parsing ignores loopback addresses and records the non-loopback guest interface address, for example enp1s0 on 192.168.86.0/24.
- The guest is shut down cleanly before export.
- qcow2 export copy exists.
- VMDK conversion completes.
- qemu-img info works for both qcow2 and VMDK.
- SHA256SUMS exists for both exported disk files.
- libvirt XML is captured.
- notes.md describes the exact process and any deviations.
- ESXi next-step import instructions are included.


Final Codex Response Requirements
---------------------------------
When finished, Codex must report:

- What was created.
- Where the VM disk lives.
- Where the export bundle lives.
- The VM LAN IP address, excluding loopback addresses.
- Whether SSH is active and tested, including the login user.
- Exact commands that failed, if any.
- Whether the VM was shut down cleanly.
- Whether ESXi was contacted.
- The contents or path of SHA256SUMS.
- The next recommended ESXi import step.

Do not claim ESXi compatibility unless the VM has actually been imported and
booted on ESXi.


Troubleshooting Quick Reference
-------------------------------
Symptom: no KVM acceleration
Likely cause: BIOS virtualization disabled or nested virtualization absent
First action: stop and report CPU virtualization evidence

Symptom: OVMF unavailable
Likely cause: ovmf package missing
First action: install ovmf if operator approves

Symptom: bridge br0 missing
Likely cause: host network not configured for bridge
First action: use NAT only if operator approves, otherwise stop

Symptom: VMDK rejected by ESXi
Likely cause: VMDK subformat mismatch
First action: record qemu-img info and retry conversion with compatible subformat

Symptom: guest does not boot on ESXi
Likely cause: EFI mismatch, controller mismatch, or bootloader issue
First action: confirm EFI firmware and try conservative disk controller

Symptom: no network on ESXi
Likely cause: adapter model changed
First action: try E1000E for first boot, then VMXNET3 after tools are active

Symptom: SSH connection refused after a successful VM boot
Likely cause: openssh-server is missing, ssh.service failed, /run/sshd is missing, or sshd_config.d contains an invalid drop-in
First action: use qemu guest agent or virt-customize to install openssh-server, create /run/sshd, validate with sshd -t, restart ssh, and verify TCP/22

Symptom: test reports guest IP as 127.0.0.1 even though LAN networking works
Likely cause: parser selected the loopback entry from virsh domifaddr before the real interface
First action: filter out 127.0.0.0/8 and require the non-loopback LAN interface address, for example 192.168.86.x on enp1s0

## Precheck Update - 2026-05-15

Before rerunning the workflow, confirm the host has every validation tool required by the SSH-access changes. The precheck on kg55 found `sshpass` and `nmap` missing; both were installed and are now required prerequisites for future tests.

Required host validation tools:

```bash
sudo apt update
sudo apt install -y sshpass nmap
command -v sshpass nmap virsh virt-install qemu-img virt-customize cloud-localds nc ip bridge brctl sha256sum curl git
```

Clean retest guidance:

- If `vm-shuttle-test01` is still defined or running, do not reuse that name for a clean retest. Use `vm-shuttle-test02`, or intentionally shut down/archive/remove the prior VM and its `/srv/vms/vm-shuttle-test01` directory first.
- Existing VMDK exports do not automatically receive live guest fixes made after export time. If SSH was repaired after a VMDK was created, shut down the VM and regenerate the VMDK from the corrected disk before treating the export as final.
- The precheck should pass with no hard failures before starting the next workflow test. Warnings about preserved prior evidence are acceptable only when the next test uses a new VM name.

Known-good precheck state from 2026-05-15:

- `libvirtd` and `virtlogd` active.
- `/dev/kvm` present and KVM modules loaded.
- `br0` up, enslaving `enp5s0f0`, with no host IPv4 address.
- Default gateway `192.168.86.1` reachable.
- Binder1 `192.168.86.33` reachable.
- Debian cloud image present at `/srv/isos/debian-stable-genericcloud-amd64.qcow2`.
- Required validation tools `sshpass` and `nmap` installed.

## Test 2 Update - 2026-05-15

During `vm-shuttle-test02`, the cloud-init `runcmd` example self-waited because it ran `cloud-init status --wait` from inside cloud-init final stage. Do not put `cloud-init status --wait` inside guest `runcmd`. Run it only from the host after SSH is reachable, or from an external validation SSH session.

Correct guest `runcmd` pattern:

```yaml
runcmd:
  - mkdir -p /run/sshd
  - install -d -m 0755 /etc/ssh/sshd_config.d
  - printf 'PasswordAuthentication yes\nKbdInteractiveAuthentication yes\n' > /etc/ssh/sshd_config.d/99-codex-password-auth.conf
  - sshd -t
  - systemctl enable --now ssh
  - systemctl enable --now qemu-guest-agent
  - systemctl enable --now open-vm-tools || true
```

Correct external validation pattern:

```bash
ssh greg@VM_LAN_IP 'cloud-init status --long || true; sudo sshd -t; systemctl is-active ssh qemu-guest-agent open-vm-tools || true'
# Do not put cloud-init wait commands in the guest runcmd/final-stage command list.
```

The `vm-shuttle-test02` export succeeded after clearing the self-wait. SSH login, `sshd -t`, qemu-guest-agent, clean shutdown, QCOW2 export, VMDK conversion, and checksums all passed.

## Root Password And `su -` Update - 2026-05-15

For lab test VMs where the operator expects `su -` to work, cloud-init must set a root password. `su -` asks for the root account password, not the invoking user's password. For this workflow, set root to the same temporary lab password as user `greg` unless the operator specifies a different password.

Cloud-init user-data should include both users in `chpasswd`:

```yaml
disable_root: false
ssh_pwauth: true
chpasswd:
  expire: false
  users:
    - name: greg
      password: LAB_PASSWORD
      type: text
    - name: root
      password: LAB_PASSWORD
      type: text
```

Validation before export:

```bash
ssh greg@VM_LAN_IP 'printf "%s\n" "LAB_PASSWORD" | su - root -c "id; hostname"'
```

This is a lab convenience setting. Do not reuse the shared root password pattern for production systems or any VM that will be exposed outside the trusted lab network.