Add Linux block device (lblk) mode documentation (experimental)

docs/architecture/concepts/linux-block-devices.md

@@ -0,0 +1,93 @@
+---
+title: "Linux Block Devices (lblk)"
+description: "How simplyblock clusters use arbitrary Linux block devices, such as SAS or SATA SSDs and cloud volumes, instead of NVMe PCIe devices."
+weight: 30650
+---
+
+{{ experimental }}
+
+Storage is onboarded by a simplyblock storage cluster as NVMe PCIe devices by default. At deployment,
+NVMe controllers are detected on the PCI bus, unbound from the kernel driver, and attached natively to
+the simplyblock storage plane container.
+
+The Linux block device mode (`lblk`) is an alternative, cluster-global device mode. In it, any Linux
+block device is accepted and no NVMe hardware is required at all: SAS or SATA SSDs behind an HBA,
+virtualized disks (virtio, Xen), or cloud volumes such as Amazon EBS.
+
+!!! warning
+    Linux block device support is experimental. It is intended for evaluation, for test environments,
+    and for deployments where NVMe devices are unavailable. For production-grade performance, local
+    NVMe devices are recommended.
+
+## Device Modes
+
+The device mode is chosen once, at cluster creation, and applies to every storage node in the
+cluster:
+
+| Mode | Storage devices | Attachment |
+|------|-----------------|------------|
+| `nvme` (default) | NVMe PCIe SSDs | SPDK native NVMe driver (kernel driver unbind) |
+| `lblk` | Any Linux disk-type block device | SPDK AIO bdev on top of the kernel block layer |
+
+In `lblk` mode, devices can be selected at deploy time by their block device name (through an allow
+list or a deny list) or by their serial number. The deployment process and cluster operations are
+otherwise identical.
+
+Because the devices remain owned by the Linux kernel in `lblk` mode, no kernel driver unbinding takes
+place, and no device is ever claimed by the simplyblock storage plane PCI layer.
+
+## Device Eligibility
+
+A block device is eligible for `lblk` onboarding if all of the following hold:
+
+- It is a whole disk (not a partition, and not a special device such as a loop, RAM, CD-ROM, or
+  device-mapper device).
+- It is not mounted, and no partition of it is mounted.
+- It is not held by another subsystem (LVM, MD RAID, or device-mapper).
+- It is not the root disk.
+- It is not read-only and reports a non-zero size.
+- It is unpartitioned. A device with an existing partition table is only accepted if it is explicitly
+  force-formatted at node addition, which wipes the partition table and all filesystem signatures.
+
+## Device Identity
+
+NVMe devices are re-identified across reboots and restarts by their PCIe address and serial number.
+Linux block device names (`/dev/sdb`, `/dev/xvdc`) are not stable across reboots, so a serial-first
+identity is used in `lblk` mode instead.
+
+The primary identity is the device serial number (or WWN), persisted in the cluster database at node
+addition and re-resolved against the live host inventory on every node restart. The stored device
+name serves only as a fallback for devices that expose no serial, and a device without any hardware
+serial is given a stable synthetic identifier at configuration time.
+
+Renaming is therefore harmless. Two disks that swap kernel names after a reboot are still matched by
+serial, and the storage stack is rebuilt on the correct disks.
+
+## Failure Detection and Handling
+
+Device failure handling in `lblk` mode is at parity with the NVMe path. IO errors on a device are
+detected by the storage stack exactly as in NVMe mode and are fed into the same device state machine:
+a device is marked unavailable after repeated errors, and a device that keeps failing is removed from
+the cluster map, with its data rebuilt from redundancy onto the remaining devices by an automatic data
+migration. A device that disappears from the host, through hot removal or a cloud volume detach, is
+detected by inventory sweeps and handled like an NVMe hot-remove event.
+
+Hung IO is handled separately. An IO timeout is enforced by the SPDK native NVMe driver, by which
+stuck IO is converted into failed IO, but AIO bdevs have no such timeout. A control-plane hung-IO
+watchdog is therefore added in `lblk` mode: a device whose IO has made no progress for a sustained
+window (30 seconds by default) is detected by queue-depth sampling and marked unavailable, entering
+the same recovery machinery. The watchdog is the safety net for devices that hang without erroring,
+because device stalls are typically converted into IO errors by kernel-level SCSI and NVMe timeouts
+well before it fires.
+
+## Restrictions
+
+The device mode is cluster-global and deploy-time only: `nvme` and `lblk` devices cannot be mixed
+within one cluster, and the mode cannot be changed after cluster creation. In `lblk` mode,
+journal-on-device deployment (a dedicated device for the journal) is required. Device partitioning is
+not supported, and neither is growing a node's device set at restart time (see
+[Linux Block Device Operations](../../non-kubernetes/operations/lblk-device-operations.md)).
+
+SMART health telemetry is not available for AIO-backed devices. Device-level performance depends on
+the underlying block device and the kernel block layer, so higher latency than with SPDK-native NVMe
+attachment is to be expected.

docs/non-kubernetes/installation/linux-block-devices.md

@@ -0,0 +1,126 @@
+---
+title: "Deploy with Linux Block Devices"
+description: "Deploying a simplyblock cluster on Linux block devices: cluster creation in lblk mode, device selection by name or serial, and node addition."
+weight: 35000
+---
+
+{{ experimental }}
+
+Deploying a storage plane on Linux block devices instead of NVMe PCIe devices follows the standard
+[storage plane installation](install-sp.md) flow. Only the differing steps are described here.
+Background on the device mode, the eligibility rules, and the device identity is found under
+[Linux Block Devices (lblk)](../../architecture/concepts/linux-block-devices.md).
+
+!!! warning
+    Linux block device support is experimental. It is intended for evaluation, for test environments,
+    and for deployments where NVMe devices are unavailable, such as cloud instances with attached
+    volumes only. For production-grade performance, local NVMe devices are recommended.
+
+## Cluster Creation
+
+The device mode is a cluster-global, deploy-time choice made when the cluster is created on the
+control plane:
+
+```bash title="Creating a cluster in lblk device mode"
+sudo {{ cliname }} cluster create --device-mode lblk
+```
+
+Two values are accepted by `--device-mode`: `nvme` (the default) and `lblk`. The mode is followed by
+all storage nodes of the cluster and cannot be changed after creation.
+
+!!! important
+    All control plane and storage node services must run a software version that supports the `lblk`
+    device mode before a cluster is created with it.
+
+## Storage Node Configuration
+
+On each storage node, devices are selected at `storage-node configure` time with the `--lblk` flag
+and an optional device selector:
+
+```bash title="Configuring a storage node with all eligible block devices"
+sudo {{ cliname }} storage-node configure --lblk --max-lvol <MAX_LOGICAL_VOLUMES>
+```
+
+Without a selector, every eligible disk on the host is used. An eligible disk is a whole, unmounted,
+unheld, and unpartitioned disk that is not the root disk (see the
+[eligibility rules](../../architecture/concepts/linux-block-devices.md#device-eligibility)).
+
+Devices can also be selected explicitly, by name or by serial number. The three selectors are
+mutually exclusive:
+
+```bash title="Selecting block devices by name"
+sudo {{ cliname }} storage-node configure \
+    --lblk --blk-names sdb,sdc --max-lvol 50
+```
+
+```bash title="Selecting all eligible block devices except some"
+sudo {{ cliname }} storage-node configure \
+    --lblk --blk-names-exclude sda --max-lvol 50
+```
+
+```bash title="Selecting block devices by serial number or WWN"
+sudo {{ cliname }} storage-node configure \
+    --lblk --blk-serials S3EVNX0M602707,S3EVNX0M602708 --max-lvol 50
+```
+
+A requested device that is busy (mounted, held, or otherwise ineligible) is an error. The
+configuration fails rather than silently skipping the device.
+
+The selected devices are stored in the resulting configuration file
+(`/etc/simplyblock/sn_config_file`) with their name, serial, stable by-id path, size, and NUMA
+assignment. As in NVMe mode, the file can be reviewed and manually edited before deployment, for
+example to remove a device from the selection.
+
+### Partitioned Devices
+
+A device carrying a partition table is not eligible by default. To reuse such a device, it is marked
+eligible with `--force` at configuration time:
+
+```bash title="Including a partitioned block device in the selection"
+sudo {{ cliname }} storage-node configure \
+    --lblk --blk-names sdb --force --max-lvol 50
+```
+
+The wipe itself happens later, at node addition, where it has to be requested explicitly with
+`--force-format`. Until then, no data is touched.
+
+## Storage Node Deployment and Addition
+
+Node deployment is unchanged:
+
+```bash title="Deploying the storage node"
+sudo {{ cliname }} storage-node deploy --ifname eth0
+```
+
+Adding the node to the cluster from a control plane node is unchanged as well, apart from one
+additional flag. If partitioned devices were force-included at configuration time, with
+`--force-format` the node addition is instructed to wipe partition tables and filesystem signatures
+(`wipefs`) from those devices:
+
+```bash title="Adding the storage node while wiping partitioned devices"
+sudo {{ cliname }} storage-node add-node \
+    --force-format <CLUSTER_ID> <NODE_IP>:5000 eth0
+```
+
+!!! danger
+    `--force-format` irreversibly destroys any data on the affected devices. The device selection in
+    the node configuration has to be verified before the node is added.
+
+During node addition, each selected device is wrapped in an SPDK AIO bdev. No kernel driver unbinding
+takes place, so the devices stay visible to the host OS but must not be used by anything else. The
+smallest device is used as the journal device, as in NVMe mode with journal-on-device deployments.
+
+Everything after node addition (cluster activation, pool creation, volume provisioning, and client
+connection) is identical to an NVMe-mode cluster.
+
+## Verification
+
+After activation, the devices are listed like NVMe devices, showing the device path instead of a PCIe
+address:
+
+```bash title="Listing the storage devices of a node"
+sudo {{ cliname }} storage-node list-devices <NODE_ID>
+```
+
+On the host, the devices remain kernel-owned and continue to be shown by `lsblk`. One `aio_<serial>`
+base bdev per device is exposed by the SPDK process.

docs/non-kubernetes/operations/lblk-device-operations.md

@@ -0,0 +1,67 @@
+---
+title: "Linux Block Device Operations"
+description: "Operating lblk-mode clusters: node restarts with serial-based device resolution, adding block devices to a node, and handling failed devices."
+weight: 20090
+---
+
+{{ experimental }}
+
+Day-2 operations specific to clusters in the Linux block device mode (`lblk`) are described below.
+Concepts and deployment are described under
+[Linux Block Devices (lblk)](../../architecture/concepts/linux-block-devices.md) and
+[Deploy with Linux Block Devices](../installation/linux-block-devices.md).
+
+## Node Restarts
+
+No `lblk`-specific handling is required for a node restart. On restart, the node's configured devices
+are re-resolved serial-first against the live host inventory. Kernel device names may have changed
+across a reboot (for example, `/dev/sdb` and `/dev/sdc` swapping), but the devices are still matched
+correctly by their persisted serial numbers, and the AIO bdevs and the storage stack above them are
+rebuilt exactly as recorded in the cluster database.
+
+A configured device that is missing from the host at restart is marked removed (the same semantics
+as a missing NVMe controller), and the standard failed-device data migration is triggered.
+
+!!! info
+    The `--ssd-pcie` option of `storage-node restart`, by which new devices are added during a
+    restart, is not supported on `lblk`-mode clusters and is rejected.
+
+## Adding Devices to a Storage Node
+
+Growing a node's device set is performed by reconfiguring and re-adding the node, not at restart
+time:
+
+1. Attach the new block devices to the host.
+2. Remove the storage node from the cluster. Its data is migrated to the remaining nodes, as with any
+   [node replacement](replacing-storage-node.md).
+3. Re-run the configuration with a selection that includes the new devices:
+
+    ```bash title="Reconfiguring the node with an extended device selection"
+    sudo {{ cliname }} storage-node configure \
+       --lblk --blk-names sdb,sdc,sdd --max-lvol 50
+    ```
+
+4. Re-add the node to the cluster. The node joins with the extended device set, and data is
+   redistributed onto it by the automatic rebalancing.
+
+Cluster capacity can alternatively be extended by
+[adding a new storage node](scaling/index.md) with its own devices.
+
+## Failed Devices
+
+Device failures are handled by the same machinery as in NVMe mode. A device producing IO errors is
+marked unavailable and, after the retry budget is exhausted, marked failed. The cluster map is
+updated, and the affected data is rebuilt from redundancy by a data migration.
+
+A device whose IO hangs without erroring is caught by the `lblk` hung-IO watchdog (roughly 30 seconds
+of zero progress with outstanding IO) and driven through the same unavailable, restart, and failed
+path. A device that disappears from the host, through hot removal or a cloud volume detach, is
+detected and treated like an NVMe hot-remove.
+
+A failed device is replaced by attaching a replacement device to the host and then following the
+[Adding Devices](#adding-devices-to-a-storage-node) procedure. Alternatively, the whole node is
+replaced, following [Replacing a Storage Node](replacing-storage-node.md).
+
+!!! info
+    SMART health information is not available for AIO-backed devices. Device health checks
+    (`storage-node check-device`) are limited to liveness and IO statistics.