Failed tempest tests in tempest.api.image.v2

[berendt]

+ echo
+ echo '## IMAGE (API)'
+ echo
+ _tempest tempest.api.image.v2
+ local regex=tempest.api.image.v2
+ docker run --rm -v /opt/tempest:/tempest -v /etc/ssl/certs:/etc/ssl/certs:ro -e PYTHONWARNINGS=ignore::SyntaxWarning --network host --name tempest registry.osism.tech/osism/tempest:latest run --workspace-path /tempest/workspace.yaml --workspace tempest --exclude-list /tempest/exclude.lst --regex tempest.api.image.v2 --concurrency 16
...

{9} tempest.api.image.v2.test_images_dependency.ImageDependencyTests.test_image_volume_server_snapshot_dependency [31.116870s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/local/lib/python3.13/site-packages/tempest/common/utils/__init__.py", line 65, in wrapper
    return f(*func_args, **func_kwargs)

      File "/usr/local/lib/python3.13/site-packages/tempest/api/image/v2/test_images_dependency.py", line 130, in test_image_volume_server_snapshot_dependency
    self.client.delete_image(base_image_id)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/services/image/v2/images_client.py", line 90, in delete_image
    resp, _ = self.delete(url)
              ~~~~~~~~~~~^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 359, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 762, in request
    self._error_checker(resp, resp_body)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 892, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)

    tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image 8a3e2f24-7472-4feb-91ce-ca6e963eb154 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

{9} tempest.api.image.v2.test_images_dependency.ImageDependencyTests.test_nova_image_snapshot_dependency [27.077120s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/usr/local/lib/python3.13/site-packages/tempest/common/utils/__init__.py", line 65, in wrapper
    return f(*func_args, **func_kwargs)

      File "/usr/local/lib/python3.13/site-packages/tempest/api/image/v2/test_images_dependency.py", line 107, in test_nova_image_snapshot_dependency
    self.client.delete_image(base_image_id)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/services/image/v2/images_client.py", line 90, in delete_image
    resp, _ = self.delete(url)
              ~~~~~~~~~~~^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 359, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 762, in request
    self._error_checker(resp, resp_body)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

      File "/usr/local/lib/python3.13/site-packages/tempest/lib/common/rest_client.py", line 892, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)

    tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image cda0d9a3-2bbf-41dc-995c-2abcb096ef81 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

[berendt]

Analysis

The two failing cases come from tempest/api/image/v2/test_images_dependency.py, which was added upstream in June/July 2024 (commits 1be8cabb46, ea914a22e2). They first reach the testbed via the recent Tempest bumps in osism/container-images:

• 79aca61 — Tempest 45 → 46.0.0
• d280afe — Tempest 46.0.0 → 46.2.0 (pulled in for NoVNCValidateMixin required by the latest ironic-tempest-plugin)

The error message

Image … could not be deleted because it is in use through the backend store outside of Glance.
is exactly the scenario these tests were written for: Glance + Cinder + Nova all on Ceph/RBD, where COW clones of the Glance image keep RBD children alive and block deletion of the parent image.

The tests are gated by [volume-feature-enabled] enable_volume_image_dep_tests in tempest.conf. They only run when this is True, so the testbed currently asserts the dependency-free behavior without the backends being configured for it.

Fix

Either disable the gate (and exclude the tests) or enable flattening on all three sides.

Option A — Skip the tests

Set in tempest.conf:

[volume-feature-enabled]
enable_volume_image_dep_tests = False

or add the two tests to exclude.lst:

tempest\.api\.image\.v2\.test_images_dependency\.ImageDependencyTests\.test_nova_image_snapshot_dependency
tempest\.api\.image\.v2\.test_images_dependency\.ImageDependencyTests\.test_image_volume_server_snapshot_dependency

This is the right choice if we deliberately keep COW clones for storage efficiency.

Option B — Make the tests pass

Required config changes:

Cinder (cinder.conf, in the RBD backend section):

[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_flatten_volume_from_snapshot = True   # default: False
rbd_max_clone_depth = 5

rbd_flatten_volume_from_snapshot = True is the actual switch for test_image_volume_server_snapshot_dependency: the volume cloned from the Glance image is flattened immediately, so the Glance image has no remaining RBD children and can be deleted.

Glance (glance-api.conf) — no dedicated flatten switch; what matters is that the COW path is enabled at all:

[DEFAULT]
show_image_direct_url = True
show_multiple_locations = True

[ceph]
rbd_thin_provisioning = True

Nova (nova.conf) — biggest lever for test_nova_image_snapshot_dependency. With images_type = rbd, instance root disks are RBD clones of the Glance image; without flattening, the Glance image stays pinned:

[libvirt]
images_type = rbd
images_rbd_pool = vms
flatten_images = True

Without Nova-side flattening of the root disk, test_nova_image_snapshot_dependency is not satisfiable regardless of what Cinder does.

Recommendation

Decide whether the testbed actually wants flattened clones (operational tradeoff: more storage / write amplification at create time vs. independent lifecycle of Glance images). Then apply Option A or B accordingly. The Tempest container itself is fine — the tests are upstream tests that have only become visible through the version bump.

[ideaship]

Root cause identified and fix in progress.

The failure chain: with the default Ceph compat level (luminous), Nova creates RBD clone v1 clones that hold a protection lock on the parent snapshot. When the test tries to delete the base image, glance_store's RBD driver calls unprotect_snap() — which raises ImageBusy under clone v1, before the trash_move() fallback (introduced in glance_store 4.6.0) is reached. Glance surfaces this as HTTP 409.

The fix is to raise the Ceph OSD map require-min-compat-client to mimic, which enables RBD clone v2 cluster-wide. Under clone v2 the protection lock is not used, unprotect_snap() succeeds, and deletion proceeds via trash_move(). This is a different approach from the flattening options in the analysis above: rather than copying image data into each clone at creation time, the parent image moves to Ceph trash and stays accessible to existing clones until they are gone — which is the scenario glance_store 4.6.0's trash_move() was written for.

Three PRs:

• ceph: add ceph_require_min_compat_client default variable defaults#284 — adds ceph_require_min_compat_client: mimic as an OSISM default variable
• openstack_config: set require-min-compat-client on OSD map container-image-ceph-ansible#690 — reads that variable and applies ceph osd set-require-min-compat-client during the ceph-pools lifecycle step (after the cluster is up, before Glance and Nova are deployed)
• tempest: re-enable image dependency tests #2895 — reverts the exclude.lst workaround from Exclude failing image dependency tempest tests #2885; should be merged after the above two