From 486e4c9ef8672587a4e6cc27da77a1063ec40414 Mon Sep 17 00:00:00 2001 From: Manohar Reddy Date: Fri, 7 Aug 2026 10:48:46 +0200 Subject: [PATCH 1/3] docs: document automatic volume placement (pin, load-aware, Pod co-location) Add the reference page for how simplyblock resolves a new volume's primary storage node: an explicit pin via the selected-storage-node annotation, Pod co-location via pod-affinity, and load-aware placement, in that order, with disable-smart-placement to opt a PVC out of the load-aware tier. Cross-link it from volume-migration.md and node-affinity.md, and apply the house style and quality-gate fixes (link targets, external-link attributes, em dash and semicolon removal) introduced on main. --- .../kubernetes/operations/volume-migration.md | 5 + docs/kubernetes/usage/volume-placement.md | 101 ++++++++++++++++++ .../operations/node-affinity.md | 7 ++ 3 files changed, 113 insertions(+) create mode 100644 docs/kubernetes/usage/volume-placement.md diff --git a/docs/kubernetes/operations/volume-migration.md b/docs/kubernetes/operations/volume-migration.md index 18380b60..bce77958 100644 --- a/docs/kubernetes/operations/volume-migration.md +++ b/docs/kubernetes/operations/volume-migration.md @@ -193,6 +193,11 @@ spec: auto-rebalancing. A one-shot placement hint from initial provisioning does not pin a volume. Such volumes remain eligible for rebalancing. +!!! note + Setting `latencyBenchmarkEnabled: true` also activates load-aware placement for newly created volumes, + independent of `enabled` (which only controls the continuous rebalancer above). See + [Automatic Volume Placement](../usage/volume-placement.md). + ## Volume Migration During Node Draining and Removal When a storage node is removed, the operator evacuates its volumes onto the remaining nodes before the node diff --git a/docs/kubernetes/usage/volume-placement.md b/docs/kubernetes/usage/volume-placement.md new file mode 100644 index 00000000..942d5ddb --- /dev/null +++ b/docs/kubernetes/usage/volume-placement.md @@ -0,0 +1,101 @@ +--- +title: "Automatic Volume Placement" +description: "Reference for the PVC annotations and StorageCluster fields that control which storage node becomes a new volume's primary node." +weight: 40050 +--- + +A new volume's primary storage node is resolved from up to three PVC annotations, evaluated in a +fixed order. If none apply, the storage cluster's built-in default placement is used. + +## Resolution Order + +| Order | Annotation | Set by | +|-------|---------------------------------------------------|------------------------| +| 1 | `simplyblock.io/selected-storage-node` | User | +| 2 | `simplyblock.io/placement-hint` | Operator (automatic) | +| 3 | `simplyblock.io/pod-affinity: "true"` (opt-in only) | User | +| — | *(none of the above)* | Storage cluster default placement | + +## Annotations + +| Annotation | Type | Default | Description | +|--------------------------------------------|-----------------------------|---------|----------------------------------------------------------------------------------------------------------------------------| +| `simplyblock.io/selected-storage-node` | string (storage node UUID) | — | Pins the volume to a specific storage node. On a new PVC, sets the primary node directly. On a bound PVC, triggers a live migration to the new node. | +| `simplyblock.io/host-id` | string (storage node UUID) | — | Deprecated alias for `selected-storage-node`. Normalized into it automatically on admission. | +| `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | +| `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | +| `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | +| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables load-aware placement for this PVC. Does not affect an explicit pin or `pod-affinity` co-location (those are already opt-in per PVC and need no separate opt-out). | + +### `simplyblock.io/selected-storage-node` + +```bash title="Pin a new PVC to a specific storage node" +kubectl annotate pvc my-pvc -n simplyblock \ + simplyblock.io/selected-storage-node=4e53efdd-86c9-424f-940c-e437eb6a2e95 +``` + +The value must be a known storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`. +Otherwise, the PVC is rejected by a validating webhook. On an already-bound PVC, a +[migration](../operations/volume-migration.md#migrating-by-pinning-a-pvc) to the new node is triggered by +this annotation instead. + +### `simplyblock.io/pod-affinity` + +```yaml title="Co-locate a new volume with its consuming Pod" +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: my-pvc + annotations: + simplyblock.io/pod-affinity: "true" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc +``` + +| Requirement | Detail | +|------------------------------------------------|---------------------------------------------------------------------------------------------| +| StorageClass binding mode | `WaitForFirstConsumer` (see [Defining a StorageClass](storage-class.md)) | +| Supported scheduling mechanisms | `nodeSelector`, node affinity, pod affinity | +| Not supported | `spec.nodeName` set directly on the Pod ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953){:target="_blank" rel="noopener"}) | +| Multiple co-located storage nodes on one worker | One is selected at random | +| Precedence | Only applies when the volume is not already pinned or covered by a placement hint | + +### `simplyblock.io/disable-smart-placement` + +```bash title="Exclude a single PVC from automatic placement" +kubectl annotate pvc my-pvc -n simplyblock \ + simplyblock.io/disable-smart-placement=true +``` + +Suppresses load-aware placement for this PVC, regardless of cluster-wide configuration. Placement then falls +through to whatever is next in the [resolution order](#resolution-order): an explicit pin, `pod-affinity` +co-location if the PVC also requests it, or the storage cluster's default placement. + +## Load-Aware Placement + +Load-aware placement for new volumes is controlled by the same `StorageCluster` field that also feeds +[auto-rebalancing's latency benchmark](../operations/volume-migration.md#auto-rebalancing): + +| Field | Type | Default | Description | +|------------------------------------------------|--------|---------|------------------------------------------------------------------------------------------------------------| +| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.enabled` (continuous rebalancer only). | + +```yaml title="Enabling load-aware placement for new volumes" +spec: + volumeAutoPlacement: + latencyBenchmarkEnabled: true + prometheusURL: "http://prometheus.simplyblock.svc:9090" +``` + +A node is eligible when it is online, passes its health check, and is below its configured logical volume +limit. + +## Clones and Snapshot Restores + +None of the above applies to a PVC created from a `VolumeSnapshot` or another PVC (`dataSource`). A clone +or restore always uses its source volume's node. diff --git a/docs/non-kubernetes/operations/node-affinity.md b/docs/non-kubernetes/operations/node-affinity.md index a9fff339..4ce23552 100644 --- a/docs/non-kubernetes/operations/node-affinity.md +++ b/docs/non-kubernetes/operations/node-affinity.md @@ -59,3 +59,10 @@ The storage node UUID (or host id) can be found using the `{{ cliname }} storage ```bash title="List all storage nodes in a storage cluster" {{ cliname }} storage-node list --cluster-id= ``` + +!!! tip + On Kubernetes, a volume's primary node can also be pinned per-PVC with the + `simplyblock.io/selected-storage-node` annotation, and a new volume can be automatically co-located + with its consuming Pod via the `simplyblock.io/pod-affinity` annotation, independent of this page's + cluster-wide `--enable-node-affinity` setting. See + [Automatic Volume Placement](../../kubernetes/usage/volume-placement.md). From 6a4615e7d93246f8b8cee1f4cc14efde6f2829f9 Mon Sep 17 00:00:00 2001 From: "Christoph Engelbert (noctarius)" Date: Fri, 7 Aug 2026 13:37:55 +0200 Subject: [PATCH 2/3] Fixed table alignment --- docs/kubernetes/usage/volume-placement.md | 48 +++++++++++------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/kubernetes/usage/volume-placement.md b/docs/kubernetes/usage/volume-placement.md index 942d5ddb..949c110e 100644 --- a/docs/kubernetes/usage/volume-placement.md +++ b/docs/kubernetes/usage/volume-placement.md @@ -9,23 +9,23 @@ fixed order. If none apply, the storage cluster's built-in default placement is ## Resolution Order -| Order | Annotation | Set by | -|-------|---------------------------------------------------|------------------------| -| 1 | `simplyblock.io/selected-storage-node` | User | -| 2 | `simplyblock.io/placement-hint` | Operator (automatic) | -| 3 | `simplyblock.io/pod-affinity: "true"` (opt-in only) | User | -| — | *(none of the above)* | Storage cluster default placement | +| Order | Annotation | Set by | +|-------|-----------------------------------------------------|-----------------------------------| +| 1 | `simplyblock.io/selected-storage-node` | User | +| 2 | `simplyblock.io/placement-hint` | Operator (automatic) | +| 3 | `simplyblock.io/pod-affinity: "true"` (opt-in only) | User | +| — | *(none of the above)* | Storage cluster default placement | ## Annotations -| Annotation | Type | Default | Description | -|--------------------------------------------|-----------------------------|---------|----------------------------------------------------------------------------------------------------------------------------| -| `simplyblock.io/selected-storage-node` | string (storage node UUID) | — | Pins the volume to a specific storage node. On a new PVC, sets the primary node directly. On a bound PVC, triggers a live migration to the new node. | -| `simplyblock.io/host-id` | string (storage node UUID) | — | Deprecated alias for `selected-storage-node`. Normalized into it automatically on admission. | -| `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | -| `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | -| `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | -| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables load-aware placement for this PVC. Does not affect an explicit pin or `pod-affinity` co-location (those are already opt-in per PVC and need no separate opt-out). | +| Annotation | Type | Default | Description | +|------------------------------------------|----------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `simplyblock.io/selected-storage-node` | string (storage node UUID) | — | Pins the volume to a specific storage node. On a new PVC, sets the primary node directly. On a bound PVC, triggers a live migration to the new node. | +| `simplyblock.io/host-id` | string (storage node UUID) | — | Deprecated alias for `selected-storage-node`. Normalized into it automatically on admission. | +| `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | +| `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | +| `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | +| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables load-aware placement for this PVC. Does not affect an explicit pin or `pod-affinity` co-location (those are already opt-in per PVC and need no separate opt-out). | ### `simplyblock.io/selected-storage-node` @@ -57,13 +57,13 @@ spec: storageClassName: simplyblock-csi-sc ``` -| Requirement | Detail | -|------------------------------------------------|---------------------------------------------------------------------------------------------| -| StorageClass binding mode | `WaitForFirstConsumer` (see [Defining a StorageClass](storage-class.md)) | -| Supported scheduling mechanisms | `nodeSelector`, node affinity, pod affinity | -| Not supported | `spec.nodeName` set directly on the Pod ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953){:target="_blank" rel="noopener"}) | -| Multiple co-located storage nodes on one worker | One is selected at random | -| Precedence | Only applies when the volume is not already pinned or covered by a placement hint | +| Requirement | Detail | +|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| StorageClass binding mode | `WaitForFirstConsumer` (see [Defining a StorageClass](storage-class.md)) | +| Supported scheduling mechanisms | `nodeSelector`, node affinity, pod affinity | +| Not supported | `spec.nodeName` set directly on the Pod ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953){:target="_blank" rel="noopener"}) | +| Multiple co-located storage nodes on one worker | One is selected at random | +| Precedence | Only applies when the volume is not already pinned or covered by a placement hint | ### `simplyblock.io/disable-smart-placement` @@ -81,9 +81,9 @@ co-location if the PVC also requests it, or the storage cluster's default placem Load-aware placement for new volumes is controlled by the same `StorageCluster` field that also feeds [auto-rebalancing's latency benchmark](../operations/volume-migration.md#auto-rebalancing): -| Field | Type | Default | Description | -|------------------------------------------------|--------|---------|------------------------------------------------------------------------------------------------------------| -| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.enabled` (continuous rebalancer only). | +| Field | Type | Default | Description | +|-----------------------------------------------|------|---------|--------------------------------------------------------------------------------------------------------------------------| +| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.enabled` (continuous rebalancer only). | ```yaml title="Enabling load-aware placement for new volumes" spec: From 56cf5c90f109ef200b35cbad298340f6a515f2fe Mon Sep 17 00:00:00 2001 From: "Christoph Engelbert (noctarius)" Date: Fri, 7 Aug 2026 14:01:50 +0200 Subject: [PATCH 3/3] Updated readability and removed disable-smart-placement and the deprecated annotations mentioning --- .../kubernetes/operations/volume-migration.md | 2 +- docs/kubernetes/usage/volume-placement.md | 123 +++++++++--------- 2 files changed, 62 insertions(+), 63 deletions(-) diff --git a/docs/kubernetes/operations/volume-migration.md b/docs/kubernetes/operations/volume-migration.md index bce77958..3978dbc9 100644 --- a/docs/kubernetes/operations/volume-migration.md +++ b/docs/kubernetes/operations/volume-migration.md @@ -195,7 +195,7 @@ spec: !!! note Setting `latencyBenchmarkEnabled: true` also activates load-aware placement for newly created volumes, - independent of `enabled` (which only controls the continuous rebalancer above). See + independent of `migrationEnabled` (which only controls the continuous rebalancer above). See [Automatic Volume Placement](../usage/volume-placement.md). ## Volume Migration During Node Draining and Removal diff --git a/docs/kubernetes/usage/volume-placement.md b/docs/kubernetes/usage/volume-placement.md index 949c110e..25699cbc 100644 --- a/docs/kubernetes/usage/volume-placement.md +++ b/docs/kubernetes/usage/volume-placement.md @@ -4,44 +4,75 @@ description: "Reference for the PVC annotations and StorageCluster fields that c weight: 40050 --- -A new volume's primary storage node is resolved from up to three PVC annotations, evaluated in a -fixed order. If none apply, the storage cluster's built-in default placement is used. +{{ experimental }} + +The primary storage node of a new volume is decided when the volume is provisioned. Three mechanisms can +decide it, each driven by a PVC annotation and evaluated in a fixed order. The first mechanism that applies +wins, and the remaining ones are not evaluated. If none applies, the storage cluster's built-in default +placement is used. ## Resolution Order -| Order | Annotation | Set by | -|-------|-----------------------------------------------------|-----------------------------------| -| 1 | `simplyblock.io/selected-storage-node` | User | -| 2 | `simplyblock.io/placement-hint` | Operator (automatic) | -| 3 | `simplyblock.io/pod-affinity: "true"` (opt-in only) | User | -| — | *(none of the above)* | Storage cluster default placement | +| Order | Mechanism | Annotation | Set by | Applies when | +|-------|-----------------------------------------------------|----------------------------------------|----------------------|-----------------------------------------------------------------------------------| +| 1 | [Pinning](#pinning-a-volume-to-a-storage-node) | `simplyblock.io/selected-storage-node` | User | The annotation names a storage node. | +| 2 | [Load-aware placement](#load-aware-placement) | `simplyblock.io/placement-hint` | Operator (automatic) | Load-aware placement is enabled for the cluster and not disabled for the PVC. | +| 3 | [Pod co-location](#co-locating-a-volume-with-a-pod) | `simplyblock.io/pod-affinity` | User | The annotation is set to `"true"` and the StorageClass is `WaitForFirstConsumer`. | +| — | Default placement | *(none of the above)* | — | No annotation applies. | -## Annotations +A [clone or a snapshot restore](#clones-and-snapshot-restores) is placed outside this order. -| Annotation | Type | Default | Description | -|------------------------------------------|----------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `simplyblock.io/selected-storage-node` | string (storage node UUID) | — | Pins the volume to a specific storage node. On a new PVC, sets the primary node directly. On a bound PVC, triggers a live migration to the new node. | -| `simplyblock.io/host-id` | string (storage node UUID) | — | Deprecated alias for `selected-storage-node`. Normalized into it automatically on admission. | -| `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | -| `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | -| `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | -| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables load-aware placement for this PVC. Does not affect an explicit pin or `pod-affinity` co-location (those are already opt-in per PVC and need no separate opt-out). | +## Pinning a Volume to a Storage Node -### `simplyblock.io/selected-storage-node` +On a new PVC, `simplyblock.io/selected-storage-node` sets the primary node directly. On an already-bound PVC, +a [migration](../operations/volume-migration.md#migrating-by-pinning-a-pvc) to the new node is triggered by +this annotation instead. -```bash title="Pin a new PVC to a specific storage node" +```bash title="Pinning a new PVC to a specific storage node" kubectl annotate pvc my-pvc -n simplyblock \ simplyblock.io/selected-storage-node=4e53efdd-86c9-424f-940c-e437eb6a2e95 ``` -The value must be a known storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`. -Otherwise, the PVC is rejected by a validating webhook. On an already-bound PVC, a -[migration](../operations/volume-migration.md#migrating-by-pinning-a-pvc) to the new node is triggered by -this annotation instead. +The value must be a known storage node UUID. Any other value is rejected by a validating webhook. The UUID +of a storage node is held in the `UUID` column of its `StorageNode` resource (short name `sn`). + +```bash title="Listing the storage nodes of a cluster with their UUIDs" +kubectl get storagenodes -n simplyblock +``` + +```plain title="Example output of the storage node listing" +NAME WORKER SOCKET NODEIDX UUID STATUS HEALTH AGE +simplyblock-node-mejue8 vm04.simplyblock3.localdomain 0 0 82198a36-fcbb-43e3-949c-0260bf40f0ac online true 43h +simplyblock-node-o6x20i vm03.simplyblock3.localdomain 0 0 707dd443-5d0e-470f-bdde-92f1238c4b01 online true 43h +simplyblock-node-v92jx7 vm02.simplyblock3.localdomain 0 0 114899a6-d708-499e-8051-bc9ca9713cf8 online true 43h +``` + +## Load-Aware Placement + +When load-aware placement selects a node for a new volume, that node is recorded on the PVC in the +`simplyblock.io/placement-hint` annotation. The hint is written by the operator rather than set by a user, +and it does not pin the volume. A node is eligible when it is online, passes its health check, and is below +its configured logical volume limit. + +Load-aware placement is controlled by the same `StorageCluster` field that also feeds +[auto-rebalancing's latency benchmark](../operations/volume-migration.md#auto-rebalancing): + +| Field | Type | Default | Description | +|-----------------------------------------------|------|---------|-----------------------------------------------------------------------------------------------------------------------------------| +| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.migrationEnabled` (continuous rebalancer only). | + +```yaml title="Enabling load-aware placement for new volumes" +spec: + volumeAutoPlacement: + latencyBenchmarkEnabled: true +``` -### `simplyblock.io/pod-affinity` +## Co-locating a Volume with a Pod -```yaml title="Co-locate a new volume with its consuming Pod" +`simplyblock.io/pod-affinity` places a new volume on a storage node that is co-located with the volume's +consuming Pod. It is a boolean, defaults to `false`, and is opt-in per PVC. + +```yaml title="Example of a PVC co-located with its consuming Pod" kind: PersistentVolumeClaim apiVersion: v1 metadata: @@ -57,43 +88,11 @@ spec: storageClassName: simplyblock-csi-sc ``` -| Requirement | Detail | -|-------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| StorageClass binding mode | `WaitForFirstConsumer` (see [Defining a StorageClass](storage-class.md)) | -| Supported scheduling mechanisms | `nodeSelector`, node affinity, pod affinity | -| Not supported | `spec.nodeName` set directly on the Pod ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953){:target="_blank" rel="noopener"}) | -| Multiple co-located storage nodes on one worker | One is selected at random | -| Precedence | Only applies when the volume is not already pinned or covered by a placement hint | - -### `simplyblock.io/disable-smart-placement` - -```bash title="Exclude a single PVC from automatic placement" -kubectl annotate pvc my-pvc -n simplyblock \ - simplyblock.io/disable-smart-placement=true -``` - -Suppresses load-aware placement for this PVC, regardless of cluster-wide configuration. Placement then falls -through to whatever is next in the [resolution order](#resolution-order): an explicit pin, `pod-affinity` -co-location if the PVC also requests it, or the storage cluster's default placement. - -## Load-Aware Placement - -Load-aware placement for new volumes is controlled by the same `StorageCluster` field that also feeds -[auto-rebalancing's latency benchmark](../operations/volume-migration.md#auto-rebalancing): - -| Field | Type | Default | Description | -|-----------------------------------------------|------|---------|--------------------------------------------------------------------------------------------------------------------------| -| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.enabled` (continuous rebalancer only). | - -```yaml title="Enabling load-aware placement for new volumes" -spec: - volumeAutoPlacement: - latencyBenchmarkEnabled: true - prometheusURL: "http://prometheus.simplyblock.svc:9090" -``` - -A node is eligible when it is online, passes its health check, and is below its configured logical volume -limit. +A `WaitForFirstConsumer` StorageClass is required (see [Defining a StorageClass](storage-class.md)). The +Pod's node is resolved from `nodeSelector`, node affinity, or pod affinity. Setting `spec.nodeName` directly +on the Pod is not supported +([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953){:target="_blank" rel="noopener"}). +When the Pod's node hosts more than one storage node, one of them is selected at random. ## Clones and Snapshot Restores