Skip to content

Document rolling and release Kolla image tags - #1062

Open
jklare wants to merge 3 commits into
mainfrom
docs-kolla-image-tag-namespaces
Open

Document rolling and release Kolla image tags#1062
jklare wants to merge 3 commits into
mainfrom
docs-kolla-image-tag-namespaces

Conversation

@jklare

@jklare jklare commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Documents the two Kolla image tag schemes, how to look up the image parameters
and tags of a specific OSISM version, and fixes the security advisory
remediation snippets that cannot be applied as written in a release-namespace
deployment.

Reported in osism/issues#1430.

Problem

The advisories tell operators to pin a rolling tag in
environments/kolla/images.yml:

neutron_server_tag: "2025.1"

A deployment that follows the OSISM 10 release
notes

and sets docker_namespace: kolla/release/2025.1 resolves this to
registry.osism.tech/kolla/release/2025.1/neutron-server:2025.1, which does not
exist. The pull fails with unknown: artifact ... not found.

Both settings are individually correct and documented. The combination is not,
and nothing documented that. Nothing documented either where the tags that can
be set come from, or which image parameters exist in the OSISM version actually
deployed — the guide only linked 002-images-kolla.yml at main.

What was verified, and how

Everything factual below was checked against the registry or the source. Listing
it explicitly so nothing here has to be taken on trust.

Method. All registry facts come from skopeoskopeo list-tags,
skopeo inspect and skopeo copy against docker://registry.osism.tech/....
The Harbor token endpoint and /v2/<repo>/tags/list were not queried
directly with a bearer token. skopeo needs no credentials for these
repositories, is the tool the OSISM stack itself uses (osism sync versions),
and is therefore also what the added documentation shows as the example command,
so every listing below is reproducible with a single command.

1. The two namespaces have disjoint tag sets. skopeo list-tags, re-run on
2026-08-21:

Repository Tags returned
kolla/neutron-server 2024.1, 2024.2, 2025.1, 2025.2
kolla/release/2025.1/neutron-server 26.0.3.20251208, 26.0.3.20260128, 26.0.3.20260328, 26.0.4.20260615, 26.0.6.20260814

2. Why they are disjoint. src/tag-images-with-the-version.py:221 in
container-images-kolla moves release builds into
/kolla/release/<OPENSTACK_VERSION>/ using only the computed
<version>.<build date> tag. The rolling tag is never rewritten into that
namespace, so the split is by construction, not a publishing gap. Introduced in
osism/container-images-kolla#672.

3. The rolling tag carries the OSSA-2026-032 fix. skopeo inspect docker://registry.osism.tech/kolla/neutron-server:2025.1: created 2026-08-21T01:33:57Z, org.opencontainers.image.version = 26.0.6,
de.osism.version = latest. The advisory names 26.0.6 as the fixed Epoxy
version.

4. The release namespace now carries it too, from OSISM 10.2.0 on.
10.2.0/base.yml in osism/release pinned Kolla 0.20260813.0 when the release
was prepared on 2026-08-13 and was bumped to 0.20260814.0 one day later by
osism/release#2665. Extracting
/sbom.yml from osism/kolla-ansible:0.20260814.0 (skopeo copy into a dir:,
then the layer holding it) gives versions.neutron: 26.0.6.20260814, and
skopeo inspect on
kolla/release/2025.1/neutron-server:26.0.6.20260814 reports
org.opencontainers.image.version = 26.0.6. OSISM 10.1.0 pins Kolla
0.20260328.0, from before the advisory, so it is unaffected by this and still
needs the override.

Worth flagging separately: the build 10.2.0 originally pinned,
0.20260813.0, contains no neutron images at all — its SBOM lists 120 images
and has no neutron key, where the complete builds list 133 and 60 versions.
With that SBOM, kolla_neutron_version falls back to openstack_version, so
neutron would have resolved to kolla/release/2025.1/neutron-server:2025.1,
a tag that does not exist. osism/release#2665 fixed that as well.

5. Which images each recommended tag covers. Read from
osism/defaults/all/002-images-kolla.yml at main: keystone_tag drives four
images (keystone, keystone-fernet, keystone-ssh, httpd), nova_tag
drives ten, neutron_server_tag and nova_api_tag one each.
neutron_rpc_server_image, neutron_periodic_worker_image and
neutron_ovn_maintenance_worker_image default to neutron_server_image.
nova_libvirt_tag derives from kolla_nova_libvirt_version, not nova_tag.

6. The version lookup chain in the new section. 10.2.0/base.yml pins
defaults_version: v0.20260712.0; that tag exists in osism/defaults and holds
both all/002-images-kolla.yml and all/002-images-ceph.yml, so both links
resolve.

7. Build and lint pass. yarn build succeeds with onBrokenLinks: 'throw';
the new anchor id="rolling-tags-and-release-tags" is generated and the inbound
links resolve. MegaLinter (documentation flavor: markdownlint,
markdown-table-formatter, codespell) reports no findings. The
unsupported file type SVG warnings in the build output are pre-existing and
unrelated.

Changes

  • configuration-guide/openstack/index.md — new Rolling tags and release tags section covering both schemes and the *_image override, which is the
    explanation of tag semantics asked for in the issue, plus a new Image parameters and tags of a specific OSISM version section: osism/release
    10.2.0/base.ymldefaults_version: v0.20260712.0
    002-images-kolla.yml / 002-images-ceph.yml at that tag, with skopeo list-tags examples for the tags of both namespaces.
  • release-notes/osism-10.md — states the tag scheme of the new namespace
    next to the docker_namespace change that introduced it, and warns that it
    holds no rolling tags.
  • OSSA-2026-001, -002, -005, -015, -022, -032 — each rolling-tag snippet
    gains the companion *_image override.
  • OSSA-2026-032 — additionally names OSISM 10.2.0 as the first release that
    ships the patched images, so release-namespace deployments can see that
    upgrading is now enough and the override is no longer required there.

The advisory snippets list every image the recommended tag covers, since a
partial override leaves the rest of the service pointing at a tag that does not
exist. That is why the nova_tag block in OSSA-2026-002 is ten lines.

Judgment calls, for review

These are decisions rather than verified facts, so they are the parts most worth
a second opinion:

  • Placement in OSSA-2026-015 — its snippet is already followed by a
    trust-policy warning, so this one was added after it rather than between the
    snippet and that warning.
  • Remediation scope left untouched — in OSSA-2026-002 the qemu-img path is
    reached in nova-compute, so the advisory's service-wide nova_tag is broader
    than strictly needed. Narrowing a published advisory's remediation seemed out
    of scope for a docs fix, so the scope was kept and made to work.
  • Wording of the kolla namespace as "rolling" — taken from the advisories'
    own existing phrasing ("Using rolling tags, ...") rather than newly coined.
  • The overrides were kept in OSSA-2026-032 rather than replaced by "upgrade
    to 10.2.0", since they remain the only route for 10.1.0 and earlier.

Not addressed here

The neutron-less Kolla build 0.20260813.0 and the fact that the 10.2.0
release entry pointed at it for a day are noted above but not acted on; whether
that needs anything beyond osism/release#2665 is a maintainer call.

Separately, osism sync versions --release <version> looks broken: its defaults
build the SBOM reference as
registry.osism.cloud/kolla/release/sbom:<kolla version>, a repository whose
newest tag is 0.20251130.0, while the current SBOM images live at
registry.osism.tech/kolla/release/<openstack_version>/sbom:<kolla version>.
That belongs in python-osism, not here, and is why the new documentation shows
skopeo list-tags rather than that command.

Open questions from the issue

The reporter asked three questions that this PR answers in the docs, but which
are worth confirming as correct:

  1. docker_namespace: kolla/release/2025.1 is correct and should stay.
  2. It does not need overriding wholesale, only per-image alongside a rolling tag,
    and from OSISM 10.2.0 on not even that for OSSA-2026-032.
  3. Unpinned images are unaffected, because their versions come from the
    versions.yml rendered in the osism/kolla-ansible image from its baked-in
    SBOM.

The security advisories recommend pinning a rolling image tag such as
`neutron_server_tag: "2025.1"` in `environments/kolla/images.yml`. In a
deployment that follows the OSISM 10 release notes and sets
`docker_namespace: kolla/release/2025.1`, that override resolves to
`registry.osism.tech/kolla/release/2025.1/neutron-server:2025.1`, which
does not exist, so the pull fails with `unknown: artifact ... not
found`.

The two namespaces on `registry.osism.tech` use disjoint tag schemes.
The `kolla` namespace carries the rolling tags named after an OpenStack
release, currently 2024.1, 2024.2, 2025.1 and 2025.2. The
`kolla/release/<openstack_version>` namespaces carry only immutable
`<project version>.<build date>` tags, produced by the release
retagging in `src/tag-images-with-the-version.py` of
container-images-kolla. A rolling tag therefore never exists in a
release namespace. Neither scheme was documented, and the advisories
did not mention the constraint at all.

Add a "Rolling tags and release tags" section to the OpenStack
configuration guide that describes both schemes and the `*_image`
override that pulls a single image from the `kolla` namespace. Note the
tag scheme in the OSISM 10 release notes, next to the
`docker_namespace` change that introduced it. Extend the affected
advisories, OSSA-2026-001, -002, -005, -015, -022 and -032, with the
matching `*_image` overrides.

Those overrides list every image the recommended tag covers, because a
partial override leaves the remaining images of the service pointing at
a tag that does not exist: `keystone_tag` drives four images and
`nova_tag` drives ten. `nova_libvirt_tag` is derived from
`kolla_nova_libvirt_version` rather than from `nova_tag` and is
therefore left out.

This is a documentation change only. It does not make the OSSA-2026-032
fix available to release deployments: the newest neutron-server tag in
the release namespace is 26.0.4.20260615, while the fix landed on
2026-07-29. Until a new release build is published, those deployments
can only obtain the fix through the rolling tag, which is what the
added guidance now explains how to use.

Reported in osism/issues#1430.

Flagged for security review because it corrects the remediation
instructions of six published advisories, which so far could not be
applied as written by deployments using a release namespace.

SecurityImpact
Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jan Klare <klare@osism.tech>
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

MegaLinter analysis: Success

Descriptor Linter Files Fixed Errors Max errors Warnings Elapsed time
✅ ACTION actionlint 5 0 0 0.04s
✅ JSON jsonlint 4 0 0 0.1s
✅ JSON prettier 4 0 0 0.34s
✅ JSON v8r 4 0 0 8.76s
✅ MARKDOWN markdownlint 168 0 0 2.86s
✅ MARKDOWN markdown-table-formatter 168 0 0 0.44s
✅ REPOSITORY betterleaks yes no no 0.97s
✅ REPOSITORY checkov yes no no 19.34s
✅ REPOSITORY git_diff yes no no 0.09s
✅ REPOSITORY secretlint yes no no 3.52s
✅ REPOSITORY trufflehog yes no no 3.67s
✅ SPELL codespell 178 0 0 0.67s
✅ SPELL lychee 178 0 0 18.2s
✅ YAML prettier 6 0 0 0.51s
✅ YAML v8r 6 0 0 6.9s
✅ YAML yamllint 6 0 0 0.67s

See detailed reports in MegaLinter artifacts

Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining FLAVOR_SUGGESTIONS: false)

  • Documentation: Custom Flavors
  • Command: npx mega-linter-runner@10.0.0 --custom-flavor-setup --custom-flavor-linters ACTION_ACTIONLINT,JSON_JSONLINT,JSON_V8R,JSON_PRETTIER,MARKDOWN_MARKDOWNLINT,MARKDOWN_MARKDOWN_TABLE_FORMATTER,REPOSITORY_CHECKOV,REPOSITORY_GIT_DIFF,REPOSITORY_BETTERLEAKS,REPOSITORY_SECRETLINT,REPOSITORY_TRUFFLEHOG,SPELL_LYCHEE,SPELL_CODESPELL,YAML_PRETTIER,YAML_YAMLLINT,YAML_V8R

MegaLinter is provided by OX Security
Show us your support by starring ⭐ the repository

@garloff

garloff commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Thanks for your extensive write-up!
I need to spend more time studying it in detail.

Before going down that route, I believe there are a few scenarios that users (or testers) might legitimately want to have supported and documented:

  1. Deploy the (images for the) originally released version
  2. Deploy the (images for the) originally released version with one or several cherry-picked (security/bug) fixes. We may reject this as hard to support due to combinatorial explosion in testing. (We might still want to support it technically, as this may be required for testing or for individual customer situations.)
  3. Deploy the (images for the) originally released version with all published maintenance/security/bug fixes for that version.
  4. Deploy the latest and greatest ... This may be something that can not be officially supported ... but is needed for testing purposes then.

I personally consider the scenario 3 as the most relevant and would like it to be fairly easy to achieve.

Do you agree with this list? (Anything I forgot? Anything that should not be on it?)
I can then review the docs w.r.t. those scenarios.
Maybe the tagging approach can be evolved to make things easier? That would be for the backlog then, for now documenting how it currently works is certainly good progress.

@jklare
jklare marked this pull request as ready for review August 21, 2026 16:13
jklare added 2 commits August 21, 2026 17:01
The "Image tags" section of the OpenStack configuration guide links the
list of image parameters from the main branch of osism/defaults. That
branch tracks the current development state and does not necessarily
match the version a deployment runs, so parameters that do not exist
yet in the deployed version, or that were already removed from it,
cannot be told apart from the ones that apply. The section also did not
say where the tags that can actually be set come from, which is the
registry and not a repository.

Add a subsection that walks that lookup for a specific version, with
OSISM 10.2.0 as the example. The base.yml of a release directory in
osism/release pins the version of every OSISM component; for 10.2.0 its
defaults_version parameter pins osism/defaults to v0.20260712.0.
Following that tag, all/002-images-kolla.yml lists the image parameters
of the OpenStack images and all/002-images-ceph.yml those of the Ceph
images, both linked at the pinned tag rather than at main.

Complement this with two `skopeo list-tags` examples for the Neutron
API image, one for the rolling tags in the kolla namespace and one for
the release tags in kolla/release/2025.1, so that the two tag schemes
described in the preceding subsection can be listed per image. No
credentials are required for either listing.

This is a documentation change only.

Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jan Klare <klare@osism.tech>
The advisory states that a fix will be included in upcoming OSISM
releases and leaves the image overrides as the only remediation. That
was accurate when it was written, but OSISM 10.2.0 now ships the
patched images: its entry in the release repository was updated to the
Kolla build 0.20260814.0, in which neutron-server of the
kolla/release/2025.1 namespace is tagged 26.0.6.20260814 and carries
Neutron 26.0.6, the version the upstream advisory names as fixed for
Epoxy. Readers on a release namespace therefore have no way to tell
from the advisory that upgrading is now enough and that the overrides
have become unnecessary.

Name OSISM 10.2.0 as the first release with the patched images, state
the tag it resolves to, and delimit this against OSISM 10.1.0 and
earlier, which pin Kolla builds from before this advisory (10.1.0 pins
0.20260328.0) and for which the rolling tags remain the only way to
obtain the fix.

Verified with skopeo against registry.osism.tech: the SBOM baked into
osism/kolla-ansible:0.20260814.0 resolves neutron to 26.0.6.20260814,
and the labels of that image report
org.opencontainers.image.version 26.0.6.

This is a documentation change only, it does not change the
remediation for any version that is still affected.

Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jan Klare <klare@osism.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

4 participants