How to test aarch64 templates / images on GitHub Actions

[msgilligan]

I am having trouble finding a reliable configuration on GitHub Actions for running aarch64 Lima images. I have tried to run them under aarch64 macOS, under aarch64 Ubuntu, and under x86_64 Ubuntu. I had limited success (with poor performance) using aarch64 emulation under x86_64 Ubuntu, but that solution breaks if I try to use Lima newer than 1.0.7.

Has anyone found a solution that works for Open Source (i.e. unpaid) project on GitHub?

[msgilligan]

Details on my use case:

I am creating a NixOS Lima image and template (See nixos-lima) and have written GitHub Actions that do the following:

1. Build Lima-compatible NixOS images for both x86_64 and aarch64.
2. Boot and run the NixOS images under Lima using provided templates.

The current GHA workflow file is here: https://github.com/nixos-lima/nixos-lima/blob/master/.github/workflows/build-image.yml
Here is a sample run: https://github.com/nixos-lima/nixos-lima/actions/runs/17311254497

The matrix for the test-image job looks like this:

      matrix:
        os: [ubuntu-24.04, macos-15]
        guest-arch: ['aarch64', 'x86_64']
        exclude:
          - os: 'ubuntu-24.04'      # Dumps core with Lima > 1.0.7
            guest-arch: 'aarch64'
          - os: 'macos-15'          # Does not support VZ or (hw accelerated) QEMU
            guest-arch: 'aarch64'
          - os: 'macos-15'          # Not tested, but would require emulation
            guest-arch: 'x86_64'

So the matrix is 2 x 2 ( os x architecture) but 3 of 4 boxes are excluded and only the ubuntu-24.04 x x86_64 box is active.

I have another workflow in a "sample" repo that successfully runs aarch64 in emulation under ubuntu-24.04 but it is using Lima 1.0.7, if I upgrade to Lima 1.0 or newer the QEMU instance coredumps.

[AkihiroSuda]

# Dumps core with Lima > 1.0.7

What do you see in the stack trace ?

[msgilligan]

I'm not sure where to find the stack trace. TBH, I'm not even sure from the log that it is QEMU that is dumping core.

From ha.stderr.log:

{"level":"debug","msg":"ResolveVMType: resolved VMType \"qemu\" (existing instance)","time":"2025-08-29T15:50:36Z"}
{"level":"debug","msg":"Creating iso file /home/runner/.lima/nixos/cidata.iso","time":"2025-08-29T15:50:36Z"}
{"level":"debug","msg":"Using /tmp/diskfs_iso2238293987 as workspace","time":"2025-08-29T15:50:36Z"}
{"level":"debug","msg":"OpenSSH version 9.6.1 detected, is GSSAPI supported: true","time":"2025-08-29T15:50:36Z"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2025-08-29T15:50:36Z"}
{"level":"info","msg":"hostagent socket created at /home/runner/.lima/nixos/ha.sock","time":"2025-08-29T15:50:36Z"}
{"level":"debug","msg":"QEMU version 8.2.2 detected","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"firmware candidates = [/home/runner/.local/share/qemu/edk2-aarch64-code.fd /usr/share/qemu/edk2-aarch64-code.fd /usr/bin/share/edk2-aarch64-code.fd /usr/share/edk2/aarch64/QEMU_CODE.fd /usr/share/AAVMF/AAVMF_CODE.fd /usr/share/qemu-efi-aarch64/QEMU_EFI.fd]","time":"2025-08-29T15:50:37Z"}
{"level":"info","msg":"Using system firmware (\"/usr/share/AAVMF/AAVMF_CODE.fd\")","time":"2025-08-29T15:50:37Z"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/home/runner/.lima/nixos/serial*.log\")","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"qCmd.Args: [/usr/bin/qemu-system-aarch64 -m 4096 -cpu max -machine virt,accel=tcg -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -boot order=c,splash-time=0,menu=on -drive file=/home/runner/.lima/nixos/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/home/runner/.lima/nixos/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev socket,id=net0,fd=3 -device virtio-net-pci,netdev=net0,mac=52:55:55:23:6f:66 -device virtio-rng-pci -display none -device virtio-gpu -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/home/runner/.lima/nixos/serial.sock,server=on,wait=off,logfile=/home/runner/.lima/nixos/serial.log -serial chardev:char-serial -chardev socket,id=char-serial-pci,path=/home/runner/.lima/nixos/serialp.sock,server=on,wait=off,logfile=/home/runner/.lima/nixos/serialp.log -device pci-serial,chardev=char-serial-pci -chardev socket,id=char-serial-virtio,path=/home/runner/.lima/nixos/serialv.sock,server=on,wait=off,logfile=/home/runner/.lima/nixos/serialv.log -device virtio-serial-pci,id=virtio-serial0,max_ports=1 -device virtconsole,chardev=char-serial-virtio,id=console0 -chardev socket,id=char-qmp,path=/home/runner/.lima/nixos/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-nixos -pidfile /home/runner/.lima/nixos/qemu.pid]","time":"2025-08-29T15:50:37Z"}
{"level":"info","msg":"Running in plain mode. Mounts, port forwarding, containerd, etc. will be ignored. Guest agent will not be running.","time":"2025-08-29T15:50:37Z"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 1: \"ssh\"","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/home/runner/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o GSSAPIAuthentication=no -o Ciphers=\"^[email protected],[email protected]\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/home/runner/.lima/nixos/ssh.sock\" -o ControlPersist=yes -p 37251 127.0.0.1 -- /bin/bash -c \"$(printf 'while read -r line; do [ -n \"$line\" ] \u0026\u0026 export \"$line\"; done\u003c\u003cEOF\\n$(sudo cat /mnt/lima-cidata/param.env)\\nEOF\\n/bin/bash')\"]","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 37251: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 37251: Connection refused\\r\\n\": exit status 255","time":"2025-08-29T15:50:37Z"}
{"level":"debug","msg":"qemu[stderr]: **","time":"2025-08-29T15:50:43Z"}
{"level":"debug","msg":"qemu[stderr]: ERROR:target/arm/internals.h:767:regime_is_user: code should not be reached","time":"2025-08-29T15:50:43Z"}
{"level":"debug","msg":"qemu[stdout]: Bail out! ERROR:target/arm/internals.h:767:regime_is_user: code should not be reached","time":"2025-08-29T15:50:43Z"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 1: \"ssh\"","time":"2025-08-29T15:50:47Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2025-08-29T15:50:47Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/home/runner/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o GSSAPIAuthentication=no -o Ciphers=\"^[email protected],[email protected]\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/home/runner/.lima/nixos/ssh.sock\" -o ControlPersist=yes -p 37251 127.0.0.1 -- /bin/bash -c \"$(printf 'while read -r line; do [ -n \"$line\" ] \u0026\u0026 export \"$line\"; done\u003c\u003cEOF\\n$(sudo cat /mnt/lima-cidata/param.env)\\nEOF\\n/bin/bash')\"]","time":"2025-08-29T15:50:48Z"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 37251: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 37251: Connection refused\\r\\n\": exit status 255","time":"2025-08-29T15:50:49Z"}
{"level":"info","msg":"Driver stopped due to error: \"signal: aborted (core dumped)\"","time":"2025-08-29T15:50:58Z"}
{"level":"info","msg":"Shutting down the host agent","time":"2025-08-29T15:50:58Z"}
{"level":"debug","msg":"shutting down the SSH master","time":"2025-08-29T15:50:58Z"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/home/runner/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o GSSAPIAuthentication=no -o Ciphers=\"^[email protected],[email protected]\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/home/runner/.lima/nixos/ssh.sock\" -o ControlPersist=yes -O exit -p 37251 127.0.0.1]","time":"2025-08-29T15:50:58Z"}
{"error":"failed to execute `ssh -O exit -p 37251 127.0.0.1`, out=\"Control socket connect(/home/runner/.lima/nixos/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2025-08-29T15:50:58Z"}
{"level":"info","msg":"Shutting down QEMU with the power button","time":"2025-08-29T15:50:58Z"}
{"level":"warning","msg":"Failed to remove SSH binding for port 37251","time":"2025-08-29T15:50:58Z"}
{"error":"dial unix /home/runner/.lima/nixos/qmp.sock: connect: connection refused","level":"warning","msg":"failed to open the QMP socket \"/home/runner/.lima/nixos/qmp.sock\", forcibly killing QEMU","time":"2025-08-29T15:50:58Z"}
{"level":"info","msg":"QEMU has already exited","time":"2025-08-29T15:50:58Z"}

[msgilligan]

I'm pretty sure it is QEMU that is coredumping, but I tried (apparently not hard enough) to get a log and didn't succeed. I decided to upgrade QEMU to 10.x (see my post below) and that seemed to resolve the issue.

[msgilligan]

I decided to use Nix to install Lima and QEMU to get a newer version of QEMU than is available via the default Ubuntu apt repository. (The current version in nixpkgs-unstable is 10.0.3)

WIth this change (and a few others) I am now able to test aarch64 images on GitHub. (See https://github.com/nixos-lima/nixos-lima/actions/runs/17338631721) Currently aarch64 images are only running in emulation on the ubuntu-24.04 runner (x86-64) and the current macos-15 and ubuntu-24.04-arm runners don't allow nested virtualization. (I may try to get emulated aarch64 to run on those instances, but my main need is met for now)

Update:

With two more changes I now have the full matrix [ubuntu-24.04, ubuntu-24.04-arm, macos-15] x ['aarch64', 'x86_64'] working:

• On macos-15and ubuntu-24.04-arm, use QEMU_SYSTEM_AARCH64 to override QEMU settings to -machine virt -cpu max
• Explicitly set --vm-type=qemu on macos-15 for aarch64 guest

See: nixos-lima/nixos-lima#37

[msgilligan]