Three-zone PKI (Web TLS / Client Communication / Secure Boot) + install-time customization preservation

[darksidemilk]

Splits FOG's certificate handling into three independent zones (Web TLS,
Client Communication, Secure Boot) and moves customization preservation into
installfog.sh itself. Stacked: this branch includes
customization-preservation, because the PKI work depends on its vhost
managed-block.

Continuation notes: docs/superpowers/CONTEXT-1013-pki-and-customizations.md.

The finding that drove this

.srvprivate.key is both the web vhost's TLS private key and the key
FOGBase::certDecrypt() opens on every fog-client authorize() handshake.
Confirmed on a live server by modulus comparison — it pairs with
srvpublic.crt, not ca.cert.pem.

So today, overwriting that file — an ACME renewal with --key-file,
--recreate-keys, a purchased cert dropped in place — breaks client
authentication while installing a perfectly valid certificate, with nothing in
the logs connecting the two. Pointing the vhost at a different file is safe;
overwriting FOG's is not. This is a live bug independent of the redesign.

PKI

FOG Server ROOT CA
├── FOG Web CA          → web server certificate (vhost)
├── FOG Server CA       → ca.cert.der (pinned by fog-client)
│   └── FOG Client Communication → the key certDecrypt() opens
└── FOG Secure Boot CA  → MOK.der (enrolled in firmware)
    └── FOG Project Secure Boot Signing → signs kernels, --addcert'd

Default on fresh installs. An existing server always resolves to
flat and is never switched automatically — switching one underneath a fleet
that pinned the old CA is the one thing this must not decide on its own.
--split-pki opts in, --legacy-pki opts a fresh install out; both fully
supported. Per-zone --web-ca-*/--client-ca-*/--root-ca-* let an admin
replace any single zone.

Secure Boot now enrols the issuer rather than the signer, so signing
leaves can be rotated, revoked, or issued per storage node without a firmware
trip. Landing it before Secure Boot ships in a stable release is deliberate:
with no release out there is no enrolled fleet, so the restructure is free
now and costs a MokManager visit to every machine later.

Also removes bin/setupacme.sh — FOG should read the certificate it is given,
not run an ACME client.

Customization preservation

Backup/restore moved into installfog.sh, so a bare ./installfog.sh
upgrade is protected too — previously only updatefog.sh did any of this.

• Vhost written as a FOG-managed block; anything outside the markers
  survives, so admins keep security fixes and their own directives. Ends the
  all-or-nothing -F trade-off.
• iPXE background keyed to the real FOG_IPXE_BG_FILE value, not "bg.png".
• Kernels: previous release kept as bzImage.<release> beside the new one,
  plus bounded generations and bin/restorekernel.sh.
• Kernel/init fields in the web UI are now dropdowns of what is on disk.
  Blank still means "inherit the default" and is never pre-selected.
• Optional custom.ipxe hook so PXE customization does not mean editing a
  generated file.
• Admin-supplied Secure Boot keys copied out of $webdirdest before the
  installer deletes it.

Verified on a real server

CentOS Stream 9 / Apache, across repeated installs and a full
uninstall→purge→fresh cycle: all chains verify; comm key and vhost key are
provably different keypairs; a signed kernel carries both the leaf and the
intermediate (sbverify --list); --legacy-pki reproduces the original flat
CA exactly; an existing install's CA is byte-identical after upgrade; custom
vhost block, renamed background and custom kernels all survive; FOG's own
shipped files are never reverted.

Eleven bugs were found this way and none by sandbox testing — three would
have shipped as silent data loss. The recurring cause was harnesses that
reproduced a function without reproducing the sequence that calls it.

Verified on real UEFI hardware

Secure Boot works via both enrolment routes with only the intermediate
trusted — enrolled as MOK.der through MokManager, or written into db
through the Setup Mode PK/KEK/db path. Machines boot FOG's leaf-signed
kernels.

That settles the question this restructure rested on and that no local testing
could answer: firmware and shim accept a chain terminating at the enrolled CA
rather than demanding the exact signer. A signing leaf can therefore be
rotated, revoked, or issued per storage node with no firmware trip — which is
the entire reason for enrolling the issuer instead of the signer. It also
exercises PXE boot by implication.

fog-client confirmed, with one follow-up for zazzles

fog-client authenticates correctly against a split server — it fetches the
comm certificate from the path it always has, so the separation needed no
client-side change.

At install it does add FOG Server CA — the Client Communication
intermediate — to the Windows Root store, rather than FOG Server ROOT CA.
Nothing is broken, but it is the wrong anchor and costs two things:

• Rotation. Trusting an intermediate as an anchor means replacing it
  requires re-pushing trust to every client — exactly the cost the Secure Boot
  zone just removed by enrolling the issuer.
• HTTPS by default. The client trusts only the Client zone, which does not
  sign the web certificate (the Web CA does), so the web certificate is
  untrusted. If the client trusted the root, every zone beneath it would
  validate and an all-FOG-PKI install could enable HTTPS out of the box.
  Confirmed by adding the root to the Windows trust store by hand -- HTTPS
  then works, so the mechanics are proven and only the client-side choice of
  certificate remains.

That change belongs in zazzles, not this PR. Nothing here needs to change
for it: the root is already published in the chain, and ca.cert.der keeps
carrying the intermediate for existing pinning.

Still not verified

nginx. All vhost work was verified on Apache only; the managed-block
splice and the netbootproto redirect exclusion both have nginx branches that
have never executed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YE3jeTexYEcAxWFbZcZgpZ