ignition-fetch.service orders on network.target, losing the metadata fetch race on DigitalOcean

[joshbrooks]

Bug

ignition-fetch.service is ordered After=network.target, which is reached when
NetworkManager starts, not when the network is usable. On the DigitalOcean platform
this means Ignition's first GET http://169.254.169.254/metadata/v1/user-data reliably
fires before DHCP has completed and fails with network is unreachable. In the common
case the retry ~200 ms later succeeds and nothing is noticed. When DHCP is slower than
Ignition's retry budget, the fetch fails permanently, and because the unit carries
OnFailure=emergency.target with OnFailureJobMode=isolate, the machine is stranded in
the initramfs with networking torn down — indistinguishable from dead hardware, with no
console, no journal and no sshd.

This is the same defect as #1081 (openstack: metadata fetcher may stop retrying before
network comes up). That was closed by #1098, which dropped the fetch timer for the
*stack platforms only. DigitalOcean still calls
f.FetchToBuffer(userdataUrl, resource.FetchOptions{}) with default options, so it
retains the pre-fix behaviour.

Operating System Version

Fedora CoreOS 44.20260707.3.1 (stable). Also present in 44.20260802.3.1 — I read the
unit on a host Zincati had already updated to that build and the ordering is unchanged.

The unit is unchanged on main as of this writing:

ConditionPathExists=/etc/initrd-release
DefaultDependencies=false
Before=ignition-complete.target
After=basic.target
ConditionPathExists=/run/ignition/neednet
ConditionPathExists=!/run/ignition.json
After=ignition-fetch-offline.service
Before=ignition-disks.service
OnFailure=emergency.target
OnFailureJobMode=isolate
After=network.target

Ignition Version

Ignition 2.26.0

Environment

DigitalOcean droplets, region sgp1, sizes s-4vcpu-8gb and s-1vcpu-1gb. Custom DO
image built from the official FCOS qcow2.gz artifact via the standard
POST /v2/images by-URL flow. ignition.platform.id=digitalocean. No custom kernel
arguments; IGNITION_ARGS is unset.

Expected Behavior

ignition-fetch.service waits until the network is actually usable before attempting to
reach the platform metadata service, so the fetch does not depend on winning a race
against DHCP.

Actual Behavior

The first fetch attempt fails on every boot I have been able to inspect — 6 of 6.
Success depends entirely on the retry landing after DHCP completes.

Annotated journal from a successful first boot (monotonic timestamps, public IP
redacted):

[    5.352637] Starting ignition-fetch.service - Ignition (fetch)...
[    5.398345] NetworkManager: dhcp4 (ens3): activation: beginning transaction (timeout in 90 seconds)
[    5.412212] NetworkManager: dhcp4 (ens3): state changed new lease, address=x.x.x.x, acd pending
[    5.448986] ignition[616]: GET http://169.254.169.254/metadata/v1/user-data: attempt #1
[    5.449469] ignition[616]: GET error: Get "http://169.254.169.254/metadata/v1/user-data":
                              dial tcp 169.254.169.254:80: connect: network is unreachable
[    5.557838] NetworkManager: dhcp4 (ens3): state changed new lease, address=x.x.x.x
[    5.593329] NetworkManager: manager: NetworkManager state is now CONNECTED_GLOBAL
[    5.650129] ignition[616]: GET http://169.254.169.254/metadata/v1/user-data: attempt #2
[    5.668666] ignition[616]: GET result: OK

Three independently provisioned long-lived hosts, all showing the same pattern on their
retained first-boot journals:

host	attempt #1	CONNECTED_GLOBAL	attempt #2	margin
A	5.448 (unreachable)	5.593	5.650 → OK	57 ms
B	6.047 (unreachable)	6.145	6.249 → OK	104 ms
C	5.502 (unreachable)	5.647	5.703 → OK	56 ms

The retry succeeds with only 56–104 ms of slack. Note also that NetworkManager's DHCP
budget here is 90 s, considerably longer than Ignition's retry budget — so a droplet
whose DHCP takes tens of seconds would eventually have had working networking, but
Ignition has already given up.

Impact: dead-on-arrival droplets

Measured over 22 FCOS droplets provisioned identically (minimal Ignition config, ~5–6 KB,
no remote contents.source):

arm	droplets	healthy	dead
FCOS	22	19	3
Ubuntu 24.04 (control, same region/size/account)	10	10	0

A dead droplet answers nothing — no ICMP, no port 22 — for the full 600 s probe window,
while healthy ones authenticate in 7–43 s.

Two observations that I think localise this to the guest rather than the platform:

1. DigitalOcean had assigned networking to the dead droplet. The API view of one
   failure, captured before teardown:

   status=active
   networks={"v4": [
     {"ip_address": "…", "netmask": "255.255.240.0", "gateway": "…", "type": "public"},
     {"ip_address": "10.…",  "netmask": "255.255.240.0", "gateway": "10.…", "type": "private"}]}
   actions=[{"type": "create", "status": "completed"}]
   

2. The stock Ubuntu control, which also reads 169.254.169.254 but does not fail the
   boot when it cannot, came up healthy every time.

I want to be explicit that the Ubuntu arm is underpowered: 3/22 vs 0/10 is Fisher
p ≈ 0.31, and at a 13.6 % rate there is still a ~23 % chance ten Ubuntu droplets all come
up clean by luck. It is corroboration, not proof. The load-bearing evidence is the
100 % first-attempt failure rate and the unit ordering itself.

Why the failure presents as a totally silent box

OnFailureJobMode=isolate stops all other units, including NetworkManager in the
initramfs, so the interface is deconfigured on the way into emergency.target. That
would explain why these droplets answer no ICMP rather than merely lacking sshd.
This part is inferred from the unit definition — I have not directly observed it, as
DigitalOcean's console is not reachable via the API and a failed box leaves no readable
journal.

Reproduction Steps

1. Build a DO custom image from the official FCOS qcow2.gz.
2. Create droplets with a minimal Ignition config as user_data (SSH key only is enough;
   no remote resources required).
3. On any droplet that boots, journalctl -b -o short-monotonic | grep -E 'attempt #|GET error|CONNECTED_GLOBAL'. The first attempt fails with network is unreachable every
   time.
4. Provision repeatedly to observe the tail: a minority never become reachable at all.

Other Information

The machinery for the fix appears to be present and already enabled. Listing the
FCOS initramfs:

etc/systemd/system/initrd.target.wants/nm-wait-online-initrd.service -> /usr/lib/systemd/system/nm-wait-online-initrd.service
etc/systemd/system/network-online.target.wants/nm-wait-online-initrd.service -> /usr/lib/systemd/system/nm-wait-online-initrd.service
usr/lib/systemd/system/network-online.target
usr/lib/systemd/system/nm-wait-online-initrd.service

network-online.target exists in the initramfs and nm-wait-online-initrd.service is
already pulled in by initrd.target.wants, so it runs on every boot — it is simply not
ordered before ignition-fetch.service.

Two possible directions, and I do not know which you would prefer:

1. Order the unit properly — Wants=network-online.target + After=network-online.target
   on ignition-fetch.service. This fixes every network-fetching platform at once rather
   than one at a time. In the common case it should cost nothing: it replaces the ~200 ms
   currently spent on a guaranteed-failing attempt plus a backoff. If the network never
   comes up, nm-wait-online times out, the target is reached, and the fetch fails as it
   does today — so the worst case is unchanged.

   The caveat I can see is that this assumes a network-online.target provider in the
   initramfs. That holds for FCOS, but I do not know whether it holds for every
   Ignition consumer (RHCOS, bare-metal images without NetworkManager), and I would not
   want to propose something that regresses those.

2. Extend interal/providers/*stack: drop timeout for config fetch #1098 to DigitalOcean — drop the fetch timer for the digitalocean
   provider as was done for the *stack platforms.

I have not tried to determine the current retry budget on the default path from source —
#1081 quotes ~10 attempts over ~30 s for the OpenStack fetcher, but I could not confirm
whether that is still the figure or whether it applies to DigitalOcean, so I have left
the observed behaviour to speak for itself rather than assert internals.

Happy to run further experiments — I have a harness that provisions N droplets, probes
ICMP / port 22 / SSH auth, captures the provider's own view of any failure, and tears
everything down. Also happy to open a PR for whichever direction you prefer.

---

Assisted-by: Claude (Anthropic)