UEFI PXE boot fails after upgrading to 1.5.10.2149 — HTTP-to-HTTPS redirect causes “Permission denied” with older iPXE binary

[RogerioKoglin]

UEFI PXE boot fails after upgrading to 1.5.10.2149 — HTTP-to-HTTPS redirect causes "Permission denied" with older iPXE binary

Summary

After upgrading FOG from 1.5.10.1903 to 1.5.10.2149, UEFI PXE boot stopped working on at least one client.

The binaries generated by the new installer did not boot correctly on the affected machine. After restoring the previous /tftpboot binaries, the client successfully loaded ipxe.efi, but iPXE failed when chaining to FOG with:

Permission denied

The issue was resolved only after restoring an Apache rewrite exception that allows /fog/service/ endpoints to remain accessible over HTTP.

I would like to understand whether this is:

1. A bug or regression in FOG 1.5.10.2149;
2. An incorrect DHCP/PXE configuration on my side;
3. A compatibility limitation in the affected clients' UEFI firmware;
4. Or an unsupported configuration because HTTPS is now expected for the entire iPXE flow.

Environment

• FOG server type: Normal/Master server
• Previous FOG version: 1.5.10.1903
• Current FOG version: 1.5.10.2149
• Previous database schema: 274
• Current database schema: 277
• Operating system: Debian GNU/Linux 13 (Trixie)
• Kernel: 6.12.100+deb13-amd64
• Apache: 2.4.68
• PHP: 8.4, using PHP-FPM
• MariaDB: 11.8.6
• TFTP server: tftpd-hpa
• DHCP server: External; FOG does not provide DHCP
• FOG server address: <FOG_SERVER_IP>
• FOG hostname: <FOG_HOSTNAME>
• Network interface: bond0, 802.3ad/LACP with four 1 Gbps interfaces
• Client boot mode: UEFI
• DHCP boot file used for the affected client: ipxe.efi

The external DHCP server provides the FOG server address as the next server/option 66 and ipxe.efi as the UEFI boot filename/option 67.

Upgrade procedure

The server operating system was fully upgraded before updating FOG.

FOG was then upgraded from a clean checkout of the official stable branch:

Version: 1.5.10.2149
Schema: 277
Commit: df82a454981637c82ce84c869ee676947ba7c44a

The installer completed successfully, updated the database, rebuilt the iPXE binaries, and restarted all FOG services.

After the upgrade:

• Apache was active;
• MariaDB was active;
• TFTP was active;
• NFS was active;
• All FOG services were active;
• systemctl --failed reported no failed units;
• ipxe.efi could be downloaded successfully over TFTP;
• File and directory permissions were correct.

Initial behavior with the newly generated binaries

The installer rebuilt all PXE binaries. Their hashes changed compared with the pre-upgrade files, including:

• ipxe.efi
• snponly.efi
• snp.efi
• intel.efi
• realtek.efi
• undionly.kpxe

On the affected UEFI machine, the newly generated ipxe.efi did not complete the boot process. From the client's perspective, it appeared that ipxe.efi could not be loaded or executed correctly.

I restored /tftpboot from the backup created immediately before the upgrade.

The restored UEFI binary identifies itself as:

iPXE/1.21.1+ (g5de5d)

After restoring it, the client loaded iPXE but failed with:

Permission denied

TFTP and filesystem validation

The TFTP path and files had the following permissions:

drwxr-xr-x 755 fogproject:fogproject /tftpboot
-rw-r--r-- 644 fogproject:fogproject /tftpboot/ipxe.efi
-rw-r--r-- 644 fogproject:fogproject /tftpboot/default.ipxe

The following binaries were downloaded successfully through TFTP, and the downloaded hashes matched the files on disk:

ipxe.efi
snponly.efi
undionly.kpxe

Therefore, the Permission denied message was not caused by Unix permissions or TFTP access.

Certificate validation

I also restored and validated the pre-upgrade FOG certificates.

The following checks passed:

• The FOG CA was valid;
• The server certificate was signed by that CA;
• The server certificate matched the private key;
• The server IP address was present in the certificate SAN;
• The FOG hostname was present in the certificate SAN;
• The certificate was within its validity period;
• A TLS connection using the FOG CA verified successfully;
• Apache configuration validation returned Syntax OK.

Example:

openssl verify -CAfile ca.cert.pem srvpublic.crt
srvpublic.crt: OK

A curl request using the FOG CA, without disabling TLS verification, also succeeded.

Relevant default.ipxe behavior

The restored default.ipxe chains to FOG over HTTP:

#!ipxe
set arch ${buildarch}
params
param mac0 ${net0/mac}
param arch ${arch}
param platform ${platform}
param product ${product}
param manufacturer ${product}
param ipxever ${version}
param filename ${filename}
param sysuuid ${uuid}
chain http://<FOG_SERVER_IP>/fog/service/ipxe/boot.php##params

Apache behavior after the upgrade

After the upgrade, the port 80 virtual host contained:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]
RewriteRule /management/other/ca.cert.der$ - [L]
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]

The affected client repeatedly produced this Apache access log entry:

<FOG_SERVER_IP>:80 <CLIENT_IP> - - "POST /fog/service/ipxe/boot.php HTTP/1.1" 302 ... "iPXE/1.21.1+ (g5de5d)"

There was no subsequent HTTPS request from that iPXE client.

From the client side, the result was:

Permission denied

Testing the same endpoint confirmed that Apache returned an HTTP 302 redirect:

POST http://<FOG_SERVER_IP>/fog/service/ipxe/boot.php
HTTP/1.1 302 Found
Location: https://<FOG_SERVER_IP>//fog/service/ipxe/boot.php

Notice that the generated redirect also contained a double slash before fog:

https://<FOG_SERVER_IP>//fog/service/ipxe/boot.php

I do not know whether the double slash is relevant to iPXE, but it may be worth checking.

Pre-upgrade Apache configuration

Before the upgrade, the port 80 virtual host had an additional condition:

RewriteCond %{REQUEST_URI} !^/fog/service/
RewriteCond %{HTTPS} off
RewriteRule ^/?(.*)$ https://%{HTTP_HOST}/$1 [R,L]

This prevented requests under /fog/service/ from being redirected to HTTPS.

The FOG installer replaced the virtual host and removed that condition.

Workaround

Restoring the pre-upgrade rewrite condition resolved the problem:

RewriteCond %{REQUEST_URI} !^/fog/service/
RewriteCond %{HTTPS} off
RewriteRule ^/?(.*)$ https://%{HTTP_HOST}/$1 [R,L]

After reloading Apache, the same iPXE request returned:

HTTP/1.1 200 OK

The response began correctly with:

#!ipxe
set fog-ip <FOG_SERVER_IP>
set fog-webroot fog

The client then booted successfully.

The FOG management interface remains available over HTTPS.

Questions

1. Is redirecting /fog/service/ipxe/boot.php from HTTP to HTTPS the intended behavior in FOG 1.5.10.2149?

2. Should the current stable installer preserve or generate an HTTP exception for /fog/service/ or /fog/service/ipxe/?

3. Is an older iPXE build such as iPXE/1.21.1+ (g5de5d) expected to follow this HTTP-to-HTTPS redirect?

4. Does FOG now require clients to use only the newly generated iPXE 2.x binaries for HTTPS boot?

5. For UEFI clients, should external DHCP provide ipxe.efi, snponly.efi, or another binary?

6. Could this indicate a UEFI compatibility problem with the newly generated ipxe.efi rather than an Apache configuration problem?

7. Should the redirect rule use the following form to avoid the double slash?

   RewriteRule ^/?(.*)$ https://%{HTTP_HOST}/$1 [R,L]

   Instead of:

   RewriteRule (.*) https://%{HTTP_HOST}/$1 [R,L]

8. Is excluding the entire /fog/service/ path from HTTPS redirection considered safe and supported, or should the exception be limited to /fog/service/ipxe/?

Additional observation

Before restoring the old binaries, Apache recorded successful HTTPS requests from the newer iPXE build:

iPXE/2.0.0+ (g766fa)
POST /fog/service/ipxe/boot.php HTTP/1.1 200
GET /fog/service/ipxe/bg.png HTTP/1.1 200

This suggests that the newer binary can use HTTPS successfully, while the older 1.21.1 binary fails after the HTTP redirect.

However, the newer ipxe.efi did not boot correctly on the affected UEFI client, which is why I restored the older binary.

I can provide packet captures, complete Apache virtual host files, iPXE screenshots, and hashes of the generated binaries if needed.

[mastacontrola]

Thanks for the writeup — the detail here made this much faster to pin down, and it turned up two real bugs plus a missing installer option. Taking your questions in the order that untangles them rather than 1-8.

The /fog/service/ exception never came from FOG

The FOG installer replaced the virtual host and removed that condition.

It didn't — that condition has never existed in FOG's installer. I searched every branch and the full history (git log -S 'REQUEST_URI} !^/fog/service') and there are no hits, ever. It was a local edit to the generated vhost, and the installer regenerated the file over it.

The blanket redirect has been there since April 2017 (2b8bacfed), which replaced an earlier /management/-only redirect. I also diffed the rewrite block between 1.5.10.1903 and 1.5.10.2149 — byte-identical. Nothing about the redirect changed in your upgrade.

If you want the installer to stop overwriting a vhost you maintain by hand, that's what -F / --no-vhost is for.

Why "Permission denied" appeared (Q1, Q3)

Your install is HTTPS, so this is working as designed — but the specific failure came from the rollback, not the upgrade.

configureDefaultiPXEfile() builds the chain URL from $httpproto, so a fresh HTTPS install writes chain https://.../boot.php. Your own log shows the new stack working end to end:

iPXE/2.0.0+ (g766fa)  POST /fog/service/ipxe/boot.php  200
GET /fog/service/ipxe/bg.png  200

Restoring /tftpboot from backup brought back both the old ipxe.efi (1.21.1) and the old default.ipxe, which chains over http://. Old binary + HTTP chain URL + a server you'd configured to redirect everything to HTTPS is what produced Permission denied. iPXE 1.21.1 does not follow that redirect into TLS.

So: yes, redirecting is intended when HTTPS is forced (Q1), and no, that older build should not be expected to follow it (Q3).

The actual bug: stable builds iPXE from an unpinned upstream clone (Q4, Q6)

This is the root cause of "the new binaries didn't boot", and it's ours.

utils/FOGiPXE/buildipxe.sh on stable does:

IPXEGIT="https://github.com/ipxe/ipxe"
if [[ -d ${BASE}/ipxe ]]; then
  cd ${BASE}/ipxe; git clean -fd; git reset --hard; git pull
else
  git clone ${IPXEGIT} ${BASE}/ipxe
fi

No tag, no commit, no branch. Every HTTPS install compiles whatever upstream iPXE HEAD happens to be that day. That is why all your hashes changed and the version jumped 1.21.1 → 2.0.0 with zero FOG commits touching iPXE — I checked, the src/ipxe submodule pointer is identical between your two releases.

It then copies FOG's own Makefile.housekeeping and config headers over the freshly pulled tree. Those overlays were written for the 1.21 era, and pasting them onto a 2.0.0 tree is the most likely explanation for your rebuilt ipxe.efi not booting. So this is a build problem, not a UEFI compatibility problem (Q6), and no, FOG does not require the 2.x binaries for HTTPS (Q4).

Already fixed on both maintained branches. utils/FOGiPXE is gone from dev-branch and working-1.6; both now pin a version and download prebuilt, hash- and signature-verified binaries instead of compiling from a moving upstream. Since dev-branch feeds stable, the pin reaches 1.5.x at the next stable release.

The double slash — good catch, fixed (Q7)

You were right, and your suggested form is exactly the fix.

In virtual-host context a RewriteRule pattern is matched against the URL-path including its leading slash, so (.*) captured /fog/... and the substitution emitted https://host//fog/.... The bare (.*) form is .htaccess idiom, where Apache strips the leading slash for you. Wrong here since 2017; Apache's MergeSlashes (default On since 2.4.39) normalises it on the way back in, which is why a visibly wrong Location header never became a visibly broken request.

Now RewriteRule ^/?(.*)$ https://%{HTTP_HOST}/$1 [R,L] on both maintained branches.

Excluding /fog/service/ — not something we'll ship (Q2, Q8)

Your workaround is sound for your server, and there's nothing wrong with keeping it. But we're not going to make it the default, and the reasoning matters for anyone else reading this later:

--force-https means force HTTPS. If an admin explicitly asks for that, the installer punching a permanent plaintext hole through the busiest path in the product — iPXE boot, and every fog-client check-in — would quietly contradict the thing they asked for. A security setting that silently exempts most of the traffic isn't the setting they think they enabled.

So we'd rather fix the real gap, which is that you couldn't turn HTTPS off.

New: --no-force-https

httpproto is persisted to .fogsettings. Once -S wrote https there, the installer's [[ -z $httpproto ]] && httpproto="http" default could never fire again, because .fogsettings is sourced first — and there was no flag to undo it. Re-running without -S kept forcing HTTPS, so hand-editing .fogsettings or the vhost was genuinely the only way out. That's the position you were put in.

Added on both maintained branches:

--no-force-https    Undo --force-https

On 1.6 that restores HTTP and HTTPS both served with no redirect; on the 1.5 line it restores HTTP-only serving. Either way it's a supported way back, instead of editing generated files.

Which binary for UEFI (Q5)

snponly.efi is the better default — it binds the firmware's own UEFI network protocol rather than replacing the NIC driver, so it behaves like the hardware vendor tested it. Use ipxe.efi only if you hit a NIC the firmware's own driver doesn't handle; it's the more invasive option and the one that hangs where the driver takeover fails.

Suggested path forward

1. Keep your rewrite exception for now — it's doing no harm.
2. When the next 1.5.x stable lands with the iPXE pin, re-run the installer and let it rebuild /tftpboot cleanly. The version you get will then be pinned rather than "whatever upstream shipped today".
3. If you'd rather not force HTTPS at all, re-run with --no-force-https and drop the hand-edit entirely.
4. If you do want to keep forcing HTTPS and keep a hand-maintained vhost, add -F / --no-vhost so the installer stops regenerating it.

Leaving this open to track the stable-branch iPXE pin reaching a release.