Skip to content

Migrate pinned volumes during storage node removal (user-directed placement) #297

Description

@noctarius

Situation

Volumes can be pinned to a specific storage node via the simplyblock.io/pinned-volume PVC annotation. Pinned volumes are deliberately excluded from automatic rebalancing (#130). The operator never moves them on its own, and today any non-empty annotation value simply marks the volume as "do not migrate".

Storage node removal (#131), however, must drain every volume off the node being removed, including pinned ones. A pinned volume currently cannot be migrated at all, so it blocks removal (or would strand the volume). The operator also has no basis to pick a destination. The whole point of pinning is that placement is the user's decision, not the operator's. A typical use case is to pin a volume of a virtual machine (e.g., kubevirt VM) to the same K8s worker which owns the virtual machine itself.

Solution

Node removal treats pinned volumes as a hard precondition, checked first, before any draining begins:

  • On starting a node-removal action, the operator checks whether the node being removed hosts any pinned volume.
  • If it does, the removal action is rejected (it does not start) with an event/condition naming the offending PVCs.
  • The user remediates by re-pinning each such volume to a different storage node of their choice. Re-pinning relocates the volume off the node being removed.
  • Once the node hosts no pinned volumes, the user re-issues the removal, which proceeds and auto-migrates the remaining (unpinned) volumes as usual.

The operator never moves or overrides a pinned volume. This keeps the pin contract intact and makes node removal fail fast with a clear, actionable reason instead of stalling mid-drain.

Behaviour

  • Removal partitions the node's volumes into: unpinned (auto-migrate), pinned-to-another-node (migrate to the pinned target), and pinned-to-this-node (blocked, needs user action).
  • Migration reuses the existing VolumeMigration mechanism (create → validate NVMe paths → continue → poll).
  • The chosen target is validated: it must be an online, eligible node in the cluster and not the node being removed; otherwise the operator warns and the removal waits.
  • The pin is honored going forward. After migration, auto-rebalancing continues to leave the volume where the user pinned it.

Status

Depends on #130 (volume migration same cluster), #132 (volume migration across clusters), and #131 (node removal). It builds on the pinning behavior introduced in #130.

Open questions

  • At the moment, the simplyblock.io/pinned-volume annotation's format is just "anything non-empty" means pinned (like true, yes, ...). It doesn't define the placement where it is pinned. That means the value format must be updated to either the storage node UUID or Kubernetes hostname. It may also require the cluster UUID if multiple clusters are located on the same host. Furthermore, it requires the removal-blocking UX (hard block + event until re-pinned vs. a grace/timeout).

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions