Add kolla drift-detection tool#604
Closed
ideaship wants to merge 12 commits into
Closed
Conversation
OSISM assembles each container image's version from a chain of separate
repos: openstack/kolla defines the images, container-images-kolla pins
concrete versions, and container-image-kolla-ansible consumes them through
its versions.yml.j2 template. Nothing checks that these stay in agreement,
so a service can fall out of the chain silently and only surface at deploy
or run time.
Add a local-first, plugin-based drift detector to catch that automatically:
- source: read a file or list a directory from any OSISM repo,
transparently from a local checkout (--base-dir) or, with no checkout,
from GitHub raw + API, so checks run offline or online unchanged. Reads
pinned repos and release-range refs from a local git clone via git
object reads; resolves a release to its upstream ref (treating 422 as
an absent ref).
- model: DriftEntry records one disagreement between an authoritative
source and a consumer and carries allowlist suppression plus optional
per-entry summary/remediation overrides.
- plugin protocol: a check declares the files it reads and returns the
drift it finds; the driver discovers checks from a registry.
- driver (check-kolla-drift.py): selects enabled checks (or --plugin),
runs them, applies the allowlist, renders a grouped, lifecycle-ordered
text report (or JSON), flags stale allowlist entries as a hard error,
and exits 0 (clean) / 1 (drift) / 2 (input error) for CI use.
- config: remote defaults with per-repo source overrides, base_dirs for
local reads, and releases/release_refs keys for range-aware checks.
- allowlist: known-intentional exceptions, each requiring a reason.
This lands the infrastructure only; the plugin registry is empty, so it
flags nothing on its own. The kolla checks that use it follow in the next
commits, each reviewable in isolation. Adds a kolla-drift-test tox env and
the pytest/responses test deps.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Roger Luethi <luethi@osism.tech>
Flag OSISM enable_* flags whose service is absent from upstream kolla-ansible's top-level enable-defaults across all supported releases (the service was removed or renamed upstream). Orphan = absent from the union over the supported range; per-release keys come from enablement.upstream_enable_keys (monolithic group_vars/all.yml at 2024.1/2024.2/2025.1, split group_vars/all/*.yml at 2025.2), resolved via release_to_ref. Guard an empty release range, which would otherwise report every flag as orphan. The SCOPE selects which OSISM enable flags are eligible. Ship the "explicit" scope: it flags the dead enable_X: "no" cleanup flags upstream kolla-ansible no longer defines at any supported release (freezer, murano, sahara, senlin, solum, vitrage and their horizon plugins, ironic_pxe_uefi, outward_rabbitmq, skydive), which is what drives the osism/defaults cleanup. The OSISM-invented common and kolla_operations flags have no upstream counterpart by design and are kept via the allowlist. Add the I/O-free enablement parse helpers (parse_enable_flags, truthy_enables, canon, release_range) and the upstream enable-defaults resolver. Expose orphan_ids() for reuse by kolla_orphan_config. Register and enable the plugin. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
kolla_enablement_orphan flags the enable_<service> flag of a service upstream kolla-ansible no longer defines, but osism/defaults also carries that service's companion vars (e.g. senlin_api_port, outward_rabbitmq_port) and image-definition blocks (<service>_tag, <service>_*_image/_tag). Those are equally dead, yet nothing enumerated them, so a cleanup that deleted only the enable flag left them dangling. For each genuinely dead service (an orphan per kolla_enablement_orphan, excluding the OSISM-invented ones kept via that check's allowlist) report every top-level var across osism/defaults all/*.yml whose name is the service id or begins with "<service id>_", grouped by the file it lives in. The enable flag itself is left to kolla_enablement_orphan (it never matches the "<service>_" prefix). Reuse kolla_enablement_orphan.orphan_ids() for the dead-service set and secrets_map.parse_secret_keys for the top-level key parse; match on the canon (hyphen/underscore) form. Register it second in the lifecycle PLUGINS order, right after kolla_enablement_orphan, and enable it in the default config. A registry test now guards that every registered plugin is enabled in the default config, so a plugin in PLUGINS but missing from the config can no longer silently never run. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
Detect OSISM kolla secrets with no upstream counterpart. The enablement and version-chain checks already reconcile OSISM enable flags and version pins against upstream, but stale secret vars left behind by removed services (e.g. outward_rabbitmq_password) went undetected. For each supported release the plugin compares the top-level keys of the cfg-cookiecutter kolla secrets template (environments/kolla/secrets.yml.<release>) against kolla-ansible's authoritative etc/kolla/passwords.yml at that release's resolved ref. A key present in the OSISM template but absent upstream is an orphaned secret. Both sides are release-specific, so the comparison is per release, not unioned. Keys are extracted with secrets_map.parse_secret_keys (a line regex, not a YAML load, because the template values are jinja/cookiecutter placeholders that do not parse as YAML); canon normalises the hyphen/underscore split. OSISM-invented secrets with no upstream counterpart are kept via the allowlist; a release with no OSISM template is skipped, not an error. Register it next to the sibling kolla_enablement_orphan and enable it in the default config. The current real-world baseline is clean across 2024.1-2025.2, so it serves as a regression gate. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
Flag OSISM-enabled kolla services (defaults all/099-kolla.yml enable_X truthy) that are buildable upstream at a supported release (a docker/X dir exists in openstack/kolla at the release's resolved ref) but absent from that release's OSISM build set (release latest/openstack-R.yml infrastructure_projects / openstack_projects). The docker/ universe is the scope filter, so feature flags are out of scope with no alias table. Range-aware via release_range + release_to_ref. Wire the defaults and release paths and enable the plugin. The live baseline is clean today (enable_valkey is not yet in defaults); the post-valkey gap is exercised by the synthetic fixtures. Add a per-plugin SUMMARY/REMEDIATION so the report names the file an operator must edit. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
Flag openstack/kolla docker/ services (pinned stable/2025.2) that have no versions['K'] line in the kolla-ansible template -- unwired pins that silently default. One-way: producer keys are not folded in, so blazar and masakari (present in the producer, absent from the template) still flag. Pin kolla to openstack/stable/2025.2 via sources, add the docker/ service parser and the versions.yml.j2 template parser, enable the plugin, and seed the allowlist with the ignore-tier services from the empirical stable/2025.2-vs-template delta: not-deployed services (cyborg, tacker, telegraf, zun -- zun has a kolla docker image but OSISM does not build or deploy it), build base layers (base, openstack_base), and naming/variant or supporting images with no distinct versions['K'] key (hacluster, letsencrypt, networking_baremetal, ovsdpdk, httpd, mariadb_server). Every seed entry maps to a real delta item so none is stale; valkey/blazar/ masakari are intentionally left to flag. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
A versions['K'] key the kolla-ansible template references but container-images-kolla's SBOM_IMAGE_TO_VERSION map never emits resolves to no value, so the template line silently falls back to openstack_version. The service looks pinned but is not -- an inert pin no error or log reveals, typically noticed only when the wrong version ships. Compare the two key spaces and nothing else: add the versions.yml.j2 template parser and the SBOM_IMAGE_TO_VERSION producer parser, normalise hyphen vs underscore, and deliberately ignore output-variable and image names (not keys; treating them as aliases would mask real drift). Classify each inert pin and route it to the right fix. A pin is an ADD only when OSISM actually deploys a built image (enable_X truthy in defaults AND a docker/X dir exists upstream): then the missing SBOM key should be wired. Otherwise the template line is dead and should be removed -- in the kolla-ansible template repo, not container-images-kolla. DriftEntry's optional summary/remediation carry the two distinct hints, and the add and remove buckets render as separate report blocks, each naming the file to edit. Register and enable the plugin. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
Re-home the retired check-kolla-inventory.py as a framework plugin: flag ansible inventory groups in upstream kolla-ansible multinode (pinned stable/2025.2) that are absent from the OSISM 50/51-kolla inventory in the generics repo. One-way; each flagged group's upstream members ride in `expected` so a maintainer knows what to add. Add the ansible INI inventory group parser (configparser, as the original script used), pin kolla_ansible to openstack/stable/2025.2, add the generics path, enable the plugin, and seed the allowlist with the empirical delta: 7 exact base-infra groups the OSISM overlay omits (compute, control, network, storage, monitoring, deployment, baremetal:children) plus 4 prefix not-deployed services (cyborg, tacker, telegraf, collectd, prefix covering their sub-groups). All 16 current delta items are suppressed and none is stale; the original's 8 dead IGNORE prefixes are dropped. The standalone script was never wired into CI or any tox env -- it was run by hand during inventory maintenance -- and this plugin fully subsumes it, so remove it. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Roger Luethi <luethi@osism.tech>
The check-kolla-drift.py detector existed in the repo but nothing ran
it: the kolla-drift-test tox env only exercises the unit tests, never
the detector against the live repos. Drift it is meant to catch (an
OSISM enable flag, version pin, or inventory group diverging from
upstream openstack/kolla and kolla-ansible) would therefore go
unnoticed until a deploy or gate broke.
Wire the detector into Zuul's periodic-daily pipeline:
- add a kolla-drift tox env that runs `python3 src/check-kolla-drift.py`
with no args, so every repo is read remotely from GitHub (no
checkout needed) and the configured allowlist applies. The script
exits 0 when clean, 1 on actionable drift or a stale allowlist
entry, and 2 on an input error, which maps directly onto a CI gate.
- add a generics-tox-kolla-drift job wrapping that env.
- run the job in periodic-daily, which catches drift introduced
upstream after a change merged, independent of any PR.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
The repo's yamllint Zuul job (check and periodic-daily pipelines) flagged 26 errors in the two drift config files: "too many spaces after colon" in kolla-drift-config.yml and "too many spaces after comma" in kolla-drift-allowlist.yml. Both stem from column-aligning the values for readability, which exceeds yamllint's one-space cap on the colons and commas rules. The files had never been linted before, so the violations surfaced only once the branch reached CI. Collapse the alignment padding to a single space after each colon and comma, and add the `---` document-start header to both files. This is a whitespace-only change: the parsed YAML (plugin enable flags and allowlist entries) is identical, and the unit suite still passes (146 tests). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
drift-detection
branch
2 times, most recently
from
53cdb66 to
34b3207
Compare
The drift detector shipped with per-commit slices of docs/check-kolla-drift.md spread across the framework and plugin commits. Those slices had drifted out of sync with the code: two of the seven plugins (kolla_orphan_config, kolla_secrets_orphan) were never documented at all, the "Adding a plugin" guide listed the wrong plugin protocol (omitting the required SUMMARY/REMEDIATION names and showing the obsolete allowlist.match/as_allowlisted idiom instead of allowlist.apply), and several passages referenced removed concepts (the dropped paths.* config, a "deferred" orphan signal, and a docs/findings/ file that does not exist in this repo). Plugin sections also followed no common template and were ordered inconsistently. The slices were dropped from the earlier commits; this commit adds one authoritative document in their place. It presents all seven plugins in lifecycle order (enabled -> built -> version-pinned -> deployed, matching the report) under a uniform template -- description, --plugin command, the inputs it reads, and the fix/allowlist remediation -- and documents the framework: running it, remote/--base-dir input resolution, per-repo source overrides, the allowlist (fields, exact/prefix match, stale detection), and an accurate "Adding a plugin" guide. Time-sensitive "as of writing it flags X" examples were removed so the doc does not go stale as the consumer repos are reconciled. Documentation only; no code or test change. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
drift-detection
branch
from
34b3207 to
59bccdc
Compare
The kolla-drift-test tox environment runs the pytest unit suite for the drift-detection code (tests/kolla_drift/, 146 tests, hermetic fixture-based, no network). It was defined in tox.ini but referenced by no Zuul job, so the unit tests never ran in CI. Add a generics-tox-kolla-drift-test job and wire it into the check pipeline so the unit tests gate every PR, plus periodic-daily for parity with the other jobs. The existing generics-tox-kolla-drift job (the live drift detector) stays periodic-only, which is correct: it inspects live upstream state and is not suitable as a PR gate. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
Contributor
Author
|
Replaced by osism/release#2561 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
check-kolla-drift.py, a plugin-based drift detector for the OSISM kollacontainer-image toolchain, and wires it into CI. It catches divergence between
the OSISM consumer repos (
defaults,release,generics, the container-imagerepos) and upstream
openstack/kolla/openstack/kolla-ansiblewhen it isintroduced — before it breaks a deploy or a gate.
The framework lands first (empty registry, flags nothing on its own), then seven
plugins in lifecycle order, each reviewable in isolation:
kolla_enablement_orphan— OSISM enables a service upstream no longer defineskolla_orphan_config— dead companion vars / image blocks for an orphaned servicekolla_secrets_orphan— OSISM kolla secret with no upstream counterpartkolla_enablement_build— enabled + upstream-buildable service missing from the build setkolla_version_chain_upstream— upstreamdocker/service with no version-pin linekolla_version_chain_inner— a referenced version pin that resolves to nothing (inert pin)kolla_inventory— upstream inventory group missing from the OSISM50/51-kollafiles(subsumes the retired
check-kolla-inventory.py, which is removed)Sources resolve remotely (GitHub) by default, so the detector runs anywhere
including CI;
--base-dirreads local checkouts. An allowlist marks intentionalexceptions (each requiring a reason); allowlist entries that match nothing are
reported stale. Exit codes: 0 clean / 1 drift or stale / 2 input error.
Full reference:
docs/check-kolla-drift.md.What this finds in practice
Run against the live OSISM repos, the detector surfaced a real backlog of drift;
the fixes it pointed to are now merged across the consumer repos:
kolla_enablement_orphan+kolla_orphan_config: removed 15 deadenable_*flags for services upstream kolla-ansible no longer ships, plus the ~57 dangling companion/image-definition vars they left behind (−103 lines).kolla_blazar_version/kolla_masakari_versionpins (built but never pinned — upstream-chain) and removed four dead inert pinskolla_{chrony,monasca,zookeeper,zun}_version(inner-chain).SBOM_IMAGE_TO_VERSION.kolla_inventorycheck gates: the missing neutron 2025.2 and ironic-dnsmasq groups.Across a full offline sweep it took the actionable-drift count from 81 → 3,
and the remaining 3 (the valkey items above) have since merged too. It also
self-audits its allowlist: an entry that no longer suppresses real drift is
reported as a hard error, so suppressions can't silently rot.
CI wiring
kolla-drifttox env runspython3 src/check-kolla-drift.py; agenerics-tox-kolla-driftZuul job wraps it and runs inperiodic-daily,catching drift introduced upstream after a change has merged.
🤖 Generated with Claude Code