Add Podman quadlet .image file support by ehelms · Pull Request #501 · theforeman/foremanctl

ehelms · 2026-05-08T14:00:57Z

Why are you introducing these changes? (Problem description, related links)

Container image references were embedded directly in each service role as variable defaults, making it difficult to override images for product builds, disconnected installs, or developer testing without modifying foremanctl-managed files.

Resolves: #277

What are the changes introduced in this pull request?

Image unit generation

Introduce a shared images role with deploy_image.yaml that generates a .image quadlet unit file and .image.d/ drop-in directory for each container image
Each service role gains an image.yaml task that calls the images role with its image name and tag, removing the per-role *_container_image variable defaults
Container roles now reference Image=<name>.image instead of full registry URLs — systemd ensures the image unit is active before any dependent container starts

Drop-in override hierarchy

Three tiers of override are supported without touching foremanctl-managed files:

Layer	Path	Who writes it
Vendor/RPM	`/usr/share/containers/systemd/<name>.image.d/10-product.conf`	Product RPM
ISO/archive	`/usr/share/containers/systemd/<name>.image.d/20-archive.conf`	ISO extraction
User	`/etc/containers/systemd/<name>.image.d/90-user.conf`	Operator

The last .image.d drop-in wins. registries.conf is the right tool when a private registry mirrors upstream names exactly; .image.d drop-ins are required when image names or tags change (e.g. foreman:nightly → foreman-rhel9:stream).

Authenticated registry handling

Each image unit gets a systemd service drop-in at /etc/systemd/system/<name>-image.service.d/auth.conf that sets REGISTRY_AUTH_FILE=/etc/foreman/registry-auth.json. Users with authenticated registries run podman login <registry> --authfile=/etc/foreman/registry-auth.json before deploying; podman silently ignores the file if it does not exist.

pull-images

pull-images deploys image units (respecting any vendor drop-ins already in place), then creates a temporary Policy=always drop-in (00-pull-always.conf) for each image, performs a daemon_reload, restarts all *-image.service units in parallel, and removes the drop-ins in an always: block so cleanup happens even on failure. The pull honours any .image.d overrides already present.

Migration containers converted to quadlet oneshot services

foreman-db-migrate and pulpcore-manager-migrate are now declared as quadlet .container units with Type=oneshot rather than being run as ephemeral containers via podman_container with detach: false. Ansible starts them with state: started and waits via async/async_status. This keeps migration runs in the systemd journal and makes failure state visible via systemctl status.

Tests

tests/images_test.py — core image files, .image.d/ drop-in directories, *-image service existence, auth drop-in presence and content, postgresql conditional on database_mode, foreman-proxy conditional on enabled_features
tests/iop/images_test.py — same checks for all IOP image units, gated on the iop feature marker

How to test this pull request

Deploy with ./foremanctl deploy
Confirm .image files exist under /etc/containers/systemd/ for each service and that <name>-image.service units are present
Confirm /etc/systemd/system/<name>-image.service.d/auth.conf exists for each image
Run ./foremanctl pull-images and verify images are pulled; re-run and confirm it pulls again (Policy=always enforced, not just missing)
Place a drop-in override:
```
# /etc/containers/systemd/foreman.image.d/90-user.conf
[Image]
Image=quay.io/foreman/foreman:pr-12345
```
Run systemctl daemon-reload and confirm systemctl cat foreman-image.service reflects the override
Run ./forge test and confirm images_test.py and iop/images_test.py pass

Checklist

Tests added/updated (if applicable)
Documentation updated (if applicable)

ekohl · 2026-05-08T14:35:25Z

Do you intend to resolve #277 with this?

ekohl · 2026-05-08T14:42:23Z

Add a Podman quadlet .image unit system with three-tier precedence: admin overrides (/etc/foremanctl/images.d/) > vendor/RPM overrides (/usr/share/foremanctl/images.d/) > generated defaults from Ansible variables

I'm wondering if we can leverage the built in /usr/share/containers/systemd/ somehow. It's a major change, but I've been wondering if we should ship the container definitions in an RPM and deploy them to /usr/share/containers/systemd, letting foremanctl only add overrides in /etc/containers/systemd.

Also not sure if you can use /etc/containers/systemd/foreman.image.d/override.conf to replace parts of it. If so, perhaps an RPM can ship /usr/share/containers/systemd/foreman.image.d/disconnected.conf to provide disconnected installation instructions.

ehelms · 2026-05-08T20:58:41Z

I'm wondering if we can leverage the built in /usr/share/containers/systemd/ somehow. It's a major change, but I've been wondering if we should ship the container definitions in an RPM and deploy them to /usr/share/containers/systemd, letting foremanctl only add overrides in /etc/containers/systemd.

As we stated before, tieing ourselves to an RPM laying down some base container will slow us down as any time it requires a change we have to wait for the RPM to cycle and it would split the base from any overrides. I think we would end up just deploying everything via overrides. Just feel like friction right now.

I did think about this and here is an alternative using this concept:

Updated Proposal

Place .image files alongside .container files in /etc/containers/systemd/, using quadlet's native drop-in mechanism for overrides.

Layout:

/etc/containers/systemd/
foreman.image # base, templated by foremanctl
foreman.image.d/
50-disconnected.conf # optional: disconnected registry override
90-user.conf # optional: user override

Precedence (last wins):

foreman.image -- foremanctl default
50-disconnected.conf -- disconnected layer
90-user.conf -- user layer

Numbered prefixes enforce ordering with gaps for future layers.

What this eliminates:

Custom /etc/foremanctl/images.d/ and /usr/share/foremanctl/images.d/ directories
Manual stat/copy/conditional precedence logic in deploy_image.yaml
The entire images role

What stays the same:

Container roles reference Image=foreman.image (no change)
foremanctl templates the base .image file (just changes the destination)
daemon-reload after deployment (already happening)

ekohl · 2026-05-12T08:42:05Z

Updated Proposal

This is what I had in mind.

Custom /etc/foremanctl/images.d/ and /usr/share/foremanctl/images.d/ directories

This is IMHO the big benefit.

evgeni · 2026-05-13T13:29:32Z

I like the idea!

There is an alternative approach from @bochi in #503, which is much simpler in the diff size, but requires custom code vs using native systemd/podman override options. Mostly mentioning it here as I wonder if this PR would also solve their usecase.

bochi · 2026-05-13T18:57:07Z

i like this approach too and will change my PR to only have the feature addition but no changes to image locations. looking forward to this getting merged 🙂

ianballou · 2026-05-13T21:01:11Z

Being able to override the container images easily would be really beneficial for development. If we eventually have containers built from PRs much like packit does with RPMs today, you could just drop those in. Or, when we need to test new versions of Pulp, we could simply drop that new Pulp container in (theforeman/pulp-oci-images#13).

ehelms · 2026-05-20T01:50:37Z

I'm not yet sure what change in the design is not leading to the failure. I will need to follow up with deeper investigation.

ehelms · 2026-05-28T20:35:11Z

I have reworked this PR based on testing and learning between the use of registries.conf and .image quadlets. The PR description template has been updated based on the current design. Generally, this will use of the combination of the two depending on situation and where each is applicable.

ekohl · 2026-05-29T10:40:15Z

+
+Both `registries.conf` and `.image.d` drop-ins can redirect where an image is pulled from, but they behave differently and suit different use cases.
+
+`registries.conf` applies a transparent redirect at pull time — the image is fetched from the `location` registry but stored in local storage under the original `prefix` name. This means `podman images` shows the upstream name (e.g., `quay.io/foreman/foreman:nightly`), and the `.image` quadlet continues to reference that same name. This works well when the private registry mirrors upstream image names and tags exactly.


Link to https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md as the official docs?

ekohl · 2026-05-29T10:40:32Z

+
+#### registries.conf vs .image.d drop-ins
+
+Both `registries.conf` and `.image.d` drop-ins can redirect where an image is pulled from, but they behave differently and suit different use cases.


Documenting both use cases is excellent.

ekohl · 2026-05-29T10:41:18Z

+
+`registries.conf` applies a transparent redirect at pull time — the image is fetched from the `location` registry but stored in local storage under the original `prefix` name. This means `podman images` shows the upstream name (e.g., `quay.io/foreman/foreman:nightly`), and the `.image` quadlet continues to reference that same name. This works well when the private registry mirrors upstream image names and tags exactly.
+
+`.image.d` drop-ins directly replace the `Image=` value in the quadlet unit. The image is pulled from and stored under the new reference. This is required when the image name or tag changes completely (e.g., `quay.io/foreman/foreman:nightly` → `registry.example.com/org/foreman-rhel9:stream`), since `registries.conf` cannot remap image names — only registry/namespace locations.


Should this point to https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html?

ekohl · 2026-05-29T10:42:40Z

+podman login registry.example.com --authfile=/etc/foreman/registry-auth.json
+```
+
+##### Disconnected install from ISO


ISO is a very specific distribution format.

Suggested change

##### Disconnected install from ISO

##### Disconnected install from local media

ekohl · 2026-05-29T10:54:55Z

+
 ```bash
-./foremanctl deploy
+podman login <registry> --authfile=/etc/foreman/registry-auth.json


Does the order matter or can you write it like this? Removing something from the end is easier than the middle.

Suggested change

podman login <registry> --authfile=/etc/foreman/registry-auth.json

podman login --authfile=/etc/foreman/registry-auth.json <registry>

ekohl · 2026-05-29T11:09:45Z

+
+The `foremanctl pull-images` command is an optional pre-deployment step that pulls all container images before running `foremanctl deploy`. This reduces deploy time and allows pre-staging images separately from deployment.
+
+`pull-images` deploys the `.image` unit files (making them available for quadlet to merge with any existing drop-ins from installed RPMs), then starts each `*-image.service` to perform the actual pull. To ensure mutable tags (such as `nightly`, `latest`, or `stream`) are always refreshed, `pull-images` temporarily creates a `Policy=always` drop-in before starting each service and removes it afterward, restoring `Policy=missing` for normal operation:


Temporary files feel a bit nasty because it it's interrupted then they may not be cleaned up. I don't think we need to solve it right now, but I wonder if we could somehow leverage podman auto-update to pull images.

Shall we track that in a follow up issue?

No. I previously researched it and it only updates based on the policy and there is no way to "temporarily" set the policy. While I agree temporary files are not the best, there really isn't another way to do this.

Spit-balling, you could theoretically have two separate services but then you gotta manage the drop-in for both. Another idea possibly, would be to use podman pull by printing the service and parsing the conf file for the image name, or using podman quadlet print.

I wonder if we should file a podman issue to see if there's tooling to support this.

Introduce .image quadlet units to decouple image sourcing from container definitions. Container roles now reference Image=<name>.image instead of full registry URLs, and a new images role handles deployment with a three-tier precedence model: admin overrides (/etc/containers/systemd/<name>.image.d/) > vendor RPMs (/usr/share/containers/systemd/<name>.image.d/) > generated defaults (/etc/containers/systemd/<name>.image). Resolves: theforeman#277 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ekohl · 2026-05-29T19:02:09Z

+In air-gapped environments, container images must be brought in without network access. `registries.conf` cannot express non-registry sources, but Podman can read images from local archive files. Images can be transported via USB or other local media using `podman save` / `podman load`, then referenced via drop-ins:
+
+See [podman-export(1)](https://docs.podman.io/en/latest/markdown/podman-export.1.html) for producing archive files.


If you use the example below you don't need podman load because it's implicit. I also wonder what the difference between podman save and podman export is. Perhaps drop the last part?

Suggested change

In air-gapped environments, container images must be brought in without network access. `registries.conf` cannot express non-registry sources, but Podman can read images from local archive files. Images can be transported via USB or other local media using `podman save` / `podman load`, then referenced via drop-ins:

See [podman-export(1)](https://docs.podman.io/en/latest/markdown/podman-export.1.html) for producing archive files.

In air-gapped environments, container images must be brought in without network access. `registries.conf` cannot express non-registry sources, but Podman can read images from local archive files. These images can be transported via USB or other local media, then referenced via drop-ins.

See [podman-export(1)](https://docs.podman.io/en/latest/markdown/podman-export.1.html) for producing archive files.

pr-processor Bot added the Not yet reviewed label May 8, 2026

ehelms mentioned this pull request May 8, 2026

Pulp upgrade container for development theforeman/pulp-oci-images#13

Open

1 task

evgeni mentioned this pull request May 13, 2026

allow downstream product overrides for features #503

Merged

2 tasks

ekohl reviewed May 13, 2026

View reviewed changes

Comment thread docs/developer/deployment.md Outdated

Comment thread src/roles/candlepin/tasks/main.yml

Comment thread src/roles/iop_advisor_frontend/defaults/main.yaml

Comment thread tests/images_test.py Outdated

pr-processor Bot removed the Not yet reviewed label May 13, 2026

ehelms force-pushed the image-quadlet-support branch 2 times, most recently from 9b0d041 to 2f9968f Compare May 15, 2026 20:24

ekohl approved these changes May 15, 2026

View reviewed changes

Comment thread docs/developer/deployment.md Outdated

arvind4501 reviewed May 19, 2026

View reviewed changes

Comment thread docs/developer/deployment.md Outdated

arvind4501 reviewed May 19, 2026

View reviewed changes

Comment thread src/playbooks/pull-images/pull-images.yaml

ehelms force-pushed the image-quadlet-support branch from 2f9968f to e392b79 Compare May 19, 2026 14:11