docs: failure domains, storage network multipathing, CLI migration/replication/backup, limits + consistency fixes

docs/architecture/concepts/failure-domains.md

@@ -0,0 +1,105 @@
+---
+title: "Failure Domains"
+description: "How simplyblock failure domains group storage nodes by rack, cabinet, or availability zone and constrain data, journal, and failover-path placement."
+weight: 30750
+---
+
+A failure domain groups storage nodes that share a common infrastructure dependency: a rack, a cabinet, a power
+distribution unit, or an availability zone. When failure domains are enabled, simplyblock spreads data chunks,
+journal copies, and failover paths across the domains so that the loss of one entire domain does not interrupt
+the availability of the cluster.
+
+Failure domains are identified by a non-negative integer chosen by the operator. Simplyblock does not detect the
+physical topology itself: every storage node is explicitly tagged with the id of the domain it belongs to when it
+is added to the cluster.
+
+!!! important
+    Failure-domain support is a deploy-time decision. It is enabled when the storage cluster is created and cannot
+    be switched on or off for an existing cluster. To gain the feature, a cluster must be redeployed.
+
+## What Failure Domains Protect
+
+With failure domains enabled, placement decisions consider the domain tag in four independent dimensions:
+
+1. **Data and parity chunks:** The distributed erasure coding spreads the chunks of each stripe across distinct
+   failure domains, so that a full domain outage leaves enough chunks to reconstruct all data within the configured
+   erasure coding scheme.
+2. **Journal copies:** The copies of the high-availability write journal are balanced across domains with a
+   per-domain cap, so that losing a whole domain always leaves enough journal copies to maintain the journal quorum.
+3. **Failover paths:** The secondary (and, with two parity chunks, tertiary) failover nodes of each logical volume
+   are placed in different failure domains than the primary node wherever possible.
+4. **Cluster status:** The health assessment of the cluster understands domains. Any combination of node and
+   device outages confined to a single failure domain keeps the cluster serving I/O in a degraded state instead
+   of suspending it.
+
+## The Placement Contract
+
+Placement constraints are applied in a fixed priority order: distinct hosts are a hard requirement, distinct
+failure domains are enforced next, and distinct physical labels are considered last.
+
+For the failover paths, the guaranteed invariant is:
+
+> Every logical volume store keeps **at least one failover path in a different failure domain** than its
+> primary node.
+
+With two failure domains and three paths (primary, secondary, tertiary), it is mathematically impossible to place
+all three paths in distinct domains. Simplyblock therefore guarantees at least one cross-domain failover path per
+volume store, which is enough to survive a full domain outage. The remaining paths are placed cross-domain
+wherever the topology allows it.
+
+At cluster activation, simplyblock arranges the hosts in a round-robin order across the failure domains and derives
+all secondary and tertiary assignments from this interleaved rotation. On a cluster with equally sized domains, this
+construction makes every secondary path cross-domain by design.
+
+## Balance Rules
+
+Failure-domain placement only works if the domains stay comparable in size. Simplyblock enforces this:
+
+- At **activation**, all failure domains must contain an **equal number of hosts**, and at least two domains must
+  exist.
+- During **operation**, the host count per domain may never diverge by more than **one host** (±1 rule). Adding or
+  removing a node is refused if it would unbalance the domains further.
+- Every domain must keep at least **two hosts** once the cluster holds data.
+
+A cluster with a one-host imbalance stays fully within the availability contract. Exactly one volume store then has
+a same-domain secondary path, and its tertiary path is still guaranteed to be cross-domain.
+
+!!! note
+    Balance is counted in physical hosts, not storage nodes. On multi-socket hosts running two storage nodes, both
+    nodes count as one host and must carry the same failure-domain id. Dedicated secondary nodes are not counted
+    toward the balance.
+
+## Failure Domains and Erasure Coding Schemes
+
+The number of failure domains should match the data protection goal:
+
+| Goal                                                                            | Recommendation                                                                                                |
+|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
+| Survive one full domain outage                                                  | At least `parity chunks + 1` distinct failure domains                                                         |
+| Survive one full domain outage plus one further node or drive failure elsewhere | Erasure coding scheme with two parity chunks (e.g., `1+2`, `2+2`) and at least as many domains as data chunks |
+
+The high-availability journal requires at least **four** journal copies on failure-domain clusters (instead of
+three), even with a single parity chunk. With three copies and two domains, one domain would hold two copies and
+its loss would break the journal quorum.
+
+## Domain Membership Is Immutable
+
+A host's failure domain cannot be changed while the host is part of the cluster. Moving a host between domains
+requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain id. This
+prevents accidental topology changes that would silently invalidate the placement of existing data.
+
+## Recovery Behavior
+
+Failure domains also change how the cluster recovers from large outages:
+
+- An outage confined to one domain (up to and including every node of the domain) keeps the cluster **degraded
+  but serving**. The cluster is not suspended.
+- With two parity chunks, the cluster additionally tolerates the loss of one entire domain **plus** one further
+  node or device outage in exactly one other domain.
+- When a whole domain returns from an outage (for example, after a rack power loss), its nodes are restarted **in
+  parallel** instead of strictly one-by-one, substantially shortening the recovery of large domains.
+
+For operating instructions (cluster creation, node addition, node removal, and the expansion rules), see
+[Managing Failure Domains](../../non-kubernetes/operations/failure-domains.md). For Kubernetes-based
+deployments, failure domains are assigned through the Simplyblock Operator. See the
+[Operator Reference](../../reference/operator/index.md).

docs/architecture/concepts/nvmf-security.md

@@ -68,7 +68,7 @@ enabled:
 - Host access control (`allow_any_host` is set to `false` for all volumes in the pool)
 
 ```bash title="Create Pool with DH-HMAC-CHAP Authentication"
-{{ cliname }} storage-pool add <POOL_NAME> --dhchap
+{{ cliname }} storage-pool add <POOL_NAME> <CLUSTER_ID> --dhchap
 ```
 
 ## Host Management

docs/architecture/high-availability-fault-tolerance.md

@@ -37,6 +37,11 @@ storage nodes along with parity fragments. This provides:
     outage and a concurrent node outage on another rack or drive failures on any other node with an storage
     overhead of just 50%.
 
+Clusters can be deployed with [failure domains](concepts/failure-domains.md): storage nodes are tagged with the
+rack, cabinet, or availability zone they belong to, and simplyblock spreads data chunks, journal copies, and
+failover paths across the domains. A failure-domain cluster keeps serving I/O through the outage of one entire
+domain (and, with two parity chunks, one additional node or drive failure in another domain).
+
 ### 2. Multipathing with Primary and Secondary Nodes
 
 Simplyblock supports NVMe over Fabrics (NVMe-oF) multipathing to provide path redundancy between clients and
@@ -56,6 +61,12 @@ with a parity level of _2_ (`1+2`, `2+2`, `4+2`) have two secondary paths.
 The number of secondary paths defines how many storage nodes can be lost at the same time without impacting the
 availability of the logical volume.
 
+In addition to the node-level path redundancy, simplyblock supports multipathing across independent **storage
+networks**: storage nodes attached with multiple data interfaces in separate VLANs expose every path on every
+network, multiplying the number of client connections (for example, four connections with one failover path and
+two networks, or six with two failover paths). This provides an alternative to link aggregation (LACP, MLAG).
+See [Storage Network Multipathing](../non-kubernetes/installation/storage-network-multipathing.md).
+
 ### 3. Redundant Control Plane and Storage Plane
 
 To ensure cluster-wide availability, simplyblock operates with full redundancy in both its control plane and

docs/deployment-preparation/cluster-deployment-options.md

@@ -56,12 +56,19 @@ using either option (defined per volume or storage class), but the cluster inter
 
 ### ```--qpair-count```
 
-The default number of queue pairs (sockets) per volume for an initiator (host) to connect to the
-target (server). More queue pairs per volume increase concurrency and volume performance, but require more
-server resources (RAM, CPU) and thus limit the total number of volumes per storage node. The default is 3.
-For a few very performant volumes, increase the amount. For a large number of less performant
-volumes, decrease it. More than 12 parallel connections have limited impact on overall performance. Also, the
-host requires at least one core per queue pair.
+The default number of queue pairs (sockets) per volume used within the storage cluster. More queue pairs per
+volume increase concurrency and volume performance, but require more server resources (RAM, CPU) and thus limit
+the total number of volumes per storage node. The default is 32.
+For a few very performant volumes, increase the amount. For a large number of less performant volumes, decrease
+it. The number of queue pairs per client (initiator) connection is controlled separately via
+`--client-qpair-count` (default 3). The host requires at least one core per queue pair.
+
+### ```--enable-failure-domain```
+
+Enables failure-domain anti-affinity for the cluster. Each storage node must then be added with a
+`--failure-domain` tag (rack, cabinet, or availability zone), and data, journal copies, and failover paths are
+spread across distinct failure domains. This option is deploy-time only: a cluster cannot be upgraded into the
+feature, it must be redeployed. See [Failure Domains](../architecture/concepts/failure-domains.md).
 
 ### ```--use-backup```
 

docs/deployment-preparation/hardware-requirements.md

@@ -77,7 +77,8 @@ As hyper-converged deployments have to share vCPUs, it is recommended to dedicat
 For RAM, it is required to define the maximum number of NVMe-oF subsystems per node. This depends on
 the assigned vCPUs and networking performance of the node. For each 10 GBit/s of dedicated network bandwidth
 it is recommended to use at least 3 subsystems. For each vCPU exceeding 8, it is recommended to use one additional
-subsystem. Use the lower of both values (dedicated network bandwidth, vCPUs).
+subsystem. Use the lower of both values (dedicated network bandwidth, vCPUs). A hard limit of 75 subsystems per
+node applies. See [Limits](../reference/limits.md).
 
 For storage nodes, simplyblock highly recommends DDR5 memory for optimal performance.
 
@@ -263,7 +264,7 @@ step. Low-level formating can also be executed manually.
 In production, simplyblock works with one of two options:
 
 - A **redundant network** for storage traffic (e.g., via LACP, Stacked Switches, MLAG, active/active or active/passive NICs, STP, or MSTP).
-- Two separate VLANs per node for storage traffic, connected via two separate NIC ports and switch paths, as well as configured as ***NVMe Multipathing***.
+- Two separate VLANs per node for storage traffic, connected via two separate NIC ports and switch paths, as well as configured as ***NVMe Multipathing*** (see [Storage Network Multipathing](../non-kubernetes/installation/storage-network-multipathing.md)).
   In such a setup simplyblock still recommend to provide a **redundant network for management traffic**, but it is not obligatory.
 
 For production, software-defined switches such as Linux Bridge or OVS cannot be used. An interface on top of a Linux

docs/kubernetes/installation/talos.md

@@ -36,7 +36,7 @@ Run the following command on the admin control pod to calculate the huge pages r
 ```bash title="Run the huge memory calculator"
 {{ cliname }} storage-node configure \
   --calculate-hp-only \
-  --max-lvol <MAX_LVOL> \
+  --max-subsys <MAX_SUBSYSTEMS> \
   --number-of-devices <NUMBER_OF_DEVICES>
 ```
 The following flags also affect the huge page calculation:
@@ -47,7 +47,7 @@ The following flags also affect the huge page calculation:
 
 ```plain title="Example output of huge pages calculator"
 [demo@demo ~]# {{ cliname }} storage-node configure \
-  --calculate-hp-only --max-lvol 10 --number-of-devices 4
+  --calculate-hp-only --max-subsys 10 --number-of-devices 4
 2026-02-22 22:27:47,017: 140705369632256: INFO: The required number of huge pages on this host is: 5776 (11552 MB)
 True
 ```

docs/kubernetes/operations/parallel-node-addition.md

@@ -26,7 +26,7 @@ threshold, which would cause cluster unavailability.
 
 ## Configuration
 
-Parallelism for non-FDB workers is controlled by `StorageNode.spec.maxParallelNodeAdds`.
+Parallelism for non-FDB workers is controlled by `StorageNodeSet.spec.maxParallelNodeAdds`.
 
 | Value         | Behavior                                                         |
 |---------------|------------------------------------------------------------------|

docs/kubernetes/operations/scaling/expanding-storage-cluster.md

@@ -13,16 +13,18 @@ designed to be minimal, it is still recommended to expand the cluster at times w
 full utilization.
 
 !!! info
-    Add storage nodes in **pairs** (i.e., 2, 4, 6, … nodes at a time).
-    Expansions with an odd number of nodes are **not supported**.
+    Storage nodes can be added **individually** (expansion mode, which integrates the new node by re-homing
+    existing failover paths) or **in groups**: at least two new nodes for clusters with one parity chunk, at
+    least three for clusters with two parity chunks (FTT 2). On clusters with failure domains, additional
+    balance rules apply. See [Failure Domains](../../../architecture/concepts/failure-domains.md).
 
 To add a new storage node, follow the installation steps for the chosen deployment method up to the point where nodes are added to the cluster, then continue here:
 
 - [Storage nodes in Kubernetes](../../installation/index.md)
 - [Storage nodes on Linux](../../../non-kubernetes/installation/install-sp.md)
 
 After adding the **first** new storage node, the cluster transitions to **IN_EXPANSION** and starts background rebalancing.
-Add the remaining node(s) required for the expansion (storage nodes must be added in **pairs**).
+Add the remaining node(s) required for the expansion.
 Once all newly added nodes are healthy/ready, finalize the expansion:
 
 ```bash title="Finalize cluster expansion"

docs/non-kubernetes/installation/index.md

@@ -50,9 +50,10 @@ On storage nodes, simplyblock can use either one network interface for both stor
 or separate interfaces (VLANs or subnets).
 
 !!! Important
-    It is possible to configure a storage cluster with NVMe-oF Multipathing. This requires two storage
+    It is possible to configure a storage cluster with NVMe-oF multipathing. This requires two storage
     VLANs per host, routed via separate NIC ports and switches for fault tolerance. This configuration can
     be used as an alternative to a HA networking setup with link aggregation (such as LACP, MLAG, and similar).
+    See [Storage Network Multipathing](storage-network-multipathing.md) for the setup instructions.
 
 To install simplyblock in a specific environment, these commands may have to be adopted to match its configuration.
 

docs/non-kubernetes/installation/install-sp.md

@@ -76,11 +76,15 @@ will configure one storage node per NUMA node.
 
 ```bash title="Configure the storage node"
 sudo {{ cliname }} storage-node configure \
-  --max-lvol <MAX_LOGICAL_VOLUMES>
+  --max-subsys <MAX_SUBSYSTEMS>
 ```
 
+The `--max-subsys` parameter defines the maximum number of NVMe-oF subsystems (and hence, in the default
+one-volume-per-subsystem layout, logical volumes) this node will serve. It drives the node's memory reservation
+and is capped by a hard limit of 75 subsystems per node (see [Limits](../../reference/limits.md)).
+
 ```plain title="Example output of storage node configure"
-[demo@demo-3 ~]# sudo {{ cliname }} storage-node configure --nodes-per-socket=2 --max-lvol=50
+[demo@demo-3 ~]# sudo {{ cliname }} storage-node configure --nodes-per-socket=2 --max-subsys=50
 2025-05-14 10:40:17,460: INFO: 0000:00:04.0 is already bound to nvme.
 0000:00:1e.0
 0000:00:1e.0
@@ -131,20 +135,26 @@ When all storage nodes are prepared, they can be added to the storage cluster.
 
 ```bash title="Attaching a storage node to the storage plane"
 sudo {{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
-  --journal-partition <NUM_OF_PARTITIONS> \
-  --data-nics <DATA_IF>
+  --data-nics <DATA_IF>[,<DATA_IF2>]
 ```
 
-If a separate NIC (e.g., BOND device) is used for storage traffic (no matter if in the cluster and between hosts and
-cluster nodes), the `--data-nics` parameter must be specified. In R25.10, zero or one data NICs are supported. Zero data
-NICs will utilize the management interface for all traffic.
+If separate NICs (e.g., a BOND device, or dedicated interfaces per storage VLAN) are used for storage traffic
+(no matter if in the cluster or between hosts and cluster nodes), the `--data-nics` parameter must be specified.
+Without it, the management interface carries all traffic. Multiple interfaces are given as a comma-separated
+list (e.g., `--data-nics eth1,eth2`), in which case all NVMe-oF subsystems listen on every data interface and
+connections are established once per interface. See
+[Storage Network Multipathing](storage-network-multipathing.md).
 
 !!! info
-    The number of partitions (_NUM_OF_PARTITIONS_) depends on the storage node setup. If a storage node has a
-    separate journaling device (e.g., an SLC NVMe device), the value should be zero (_0_) to prevent the storage
-    devices from being partitioned. This improves the performance and prevents device sharing between the journal and
-    the actual data storage location. However, in most cases, a separate journaling device is not available or required
-    and the value of `--journal-partition` has to be 1 (default if nothing is specified).
+    By default, simplyblock auto-creates small journal partitions on the NVMe data devices (a maximum of 3% of
+    the total available raw disk space). If a storage node has a separate journaling device (e.g., an SLC NVMe
+    device), pass `--enable-journal-device` to use the smallest NVMe device of the node exclusively for the
+    journal. This improves performance and prevents device sharing between the journal and the actual data
+    storage location.
+
+If the cluster was created with failure-domain support, every node must additionally be tagged with its
+failure-domain id via `--failure-domain <ID>`. See
+[Managing Failure Domains](../operations/failure-domains.md).
 
 The output will look something like the following example: