diff --git a/docs/architecture/concepts/linux-block-devices.md b/docs/architecture/concepts/linux-block-devices.md new file mode 100644 index 00000000..e8fb7443 --- /dev/null +++ b/docs/architecture/concepts/linux-block-devices.md @@ -0,0 +1,93 @@ +--- +title: "Linux Block Devices (lblk)" +description: "How simplyblock clusters use arbitrary Linux block devices, such as SAS or SATA SSDs and cloud volumes, instead of NVMe PCIe devices." +weight: 30650 +--- + +{{ experimental }} + +Storage is onboarded by a simplyblock storage cluster as NVMe PCIe devices by default. At deployment, +NVMe controllers are detected on the PCI bus, unbound from the kernel driver, and attached natively to +the simplyblock storage plane container. + +The Linux block device mode (`lblk`) is an alternative, cluster-global device mode. In it, any Linux +block device is accepted and no NVMe hardware is required at all: SAS or SATA SSDs behind an HBA, +virtualized disks (virtio, Xen), or cloud volumes such as Amazon EBS. + +!!! warning + Linux block device support is experimental. It is intended for evaluation, for test environments, + and for deployments where NVMe devices are unavailable. For production-grade performance, local + NVMe devices are recommended. + +## Device Modes + +The device mode is chosen once, at cluster creation, and applies to every storage node in the +cluster: + +| Mode | Storage devices | Attachment | +|------|-----------------|------------| +| `nvme` (default) | NVMe PCIe SSDs | SPDK native NVMe driver (kernel driver unbind) | +| `lblk` | Any Linux disk-type block device | SPDK AIO bdev on top of the kernel block layer | + +In `lblk` mode, devices can be selected at deploy time by their block device name (through an allow +list or a deny list) or by their serial number. The deployment process and cluster operations are +otherwise identical. + +Because the devices remain owned by the Linux kernel in `lblk` mode, no kernel driver unbinding takes +place, and no device is ever claimed by the simplyblock storage plane PCI layer. + +## Device Eligibility + +A block device is eligible for `lblk` onboarding if all of the following hold: + +- It is a whole disk (not a partition, and not a special device such as a loop, RAM, CD-ROM, or + device-mapper device). +- It is not mounted, and no partition of it is mounted. +- It is not held by another subsystem (LVM, MD RAID, or device-mapper). +- It is not the root disk. +- It is not read-only and reports a non-zero size. +- It is unpartitioned. A device with an existing partition table is only accepted if it is explicitly + force-formatted at node addition, which wipes the partition table and all filesystem signatures. + +## Device Identity + +NVMe devices are re-identified across reboots and restarts by their PCIe address and serial number. +Linux block device names (`/dev/sdb`, `/dev/xvdc`) are not stable across reboots, so a serial-first +identity is used in `lblk` mode instead. + +The primary identity is the device serial number (or WWN), persisted in the cluster database at node +addition and re-resolved against the live host inventory on every node restart. The stored device +name serves only as a fallback for devices that expose no serial, and a device without any hardware +serial is given a stable synthetic identifier at configuration time. + +Renaming is therefore harmless. Two disks that swap kernel names after a reboot are still matched by +serial, and the storage stack is rebuilt on the correct disks. + +## Failure Detection and Handling + +Device failure handling in `lblk` mode is at parity with the NVMe path. IO errors on a device are +detected by the storage stack exactly as in NVMe mode and are fed into the same device state machine: +a device is marked unavailable after repeated errors, and a device that keeps failing is removed from +the cluster map, with its data rebuilt from redundancy onto the remaining devices by an automatic data +migration. A device that disappears from the host, through hot removal or a cloud volume detach, is +detected by inventory sweeps and handled like an NVMe hot-remove event. + +Hung IO is handled separately. An IO timeout is enforced by the SPDK native NVMe driver, by which +stuck IO is converted into failed IO, but AIO bdevs have no such timeout. A control-plane hung-IO +watchdog is therefore added in `lblk` mode: a device whose IO has made no progress for a sustained +window (30 seconds by default) is detected by queue-depth sampling and marked unavailable, entering +the same recovery machinery. The watchdog is the safety net for devices that hang without erroring, +because device stalls are typically converted into IO errors by kernel-level SCSI and NVMe timeouts +well before it fires. + +## Restrictions + +The device mode is cluster-global and deploy-time only: `nvme` and `lblk` devices cannot be mixed +within one cluster, and the mode cannot be changed after cluster creation. In `lblk` mode, +journal-on-device deployment (a dedicated device for the journal) is required. Device partitioning is +not supported, and neither is growing a node's device set at restart time (see +[Linux Block Device Operations](../../non-kubernetes/operations/lblk-device-operations.md)). + +SMART health telemetry is not available for AIO-backed devices. Device-level performance depends on +the underlying block device and the kernel block layer, so higher latency than with SPDK-native NVMe +attachment is to be expected. diff --git a/docs/non-kubernetes/installation/linux-block-devices.md b/docs/non-kubernetes/installation/linux-block-devices.md new file mode 100644 index 00000000..74aa3903 --- /dev/null +++ b/docs/non-kubernetes/installation/linux-block-devices.md @@ -0,0 +1,126 @@ +--- +title: "Deploy with Linux Block Devices" +description: "Deploying a simplyblock cluster on Linux block devices: cluster creation in lblk mode, device selection by name or serial, and node addition." +weight: 35000 +--- + +{{ experimental }} + +Deploying a storage plane on Linux block devices instead of NVMe PCIe devices follows the standard +[storage plane installation](install-sp.md) flow. Only the differing steps are described here. +Background on the device mode, the eligibility rules, and the device identity is found under +[Linux Block Devices (lblk)](../../architecture/concepts/linux-block-devices.md). + +!!! warning + Linux block device support is experimental. It is intended for evaluation, for test environments, + and for deployments where NVMe devices are unavailable, such as cloud instances with attached + volumes only. For production-grade performance, local NVMe devices are recommended. + +## Cluster Creation + +The device mode is a cluster-global, deploy-time choice made when the cluster is created on the +control plane: + +```bash title="Creating a cluster in lblk device mode" +sudo {{ cliname }} cluster create --device-mode lblk +``` + +Two values are accepted by `--device-mode`: `nvme` (the default) and `lblk`. The mode is followed by +all storage nodes of the cluster and cannot be changed after creation. + +!!! important + All control plane and storage node services must run a software version that supports the `lblk` + device mode before a cluster is created with it. + +## Storage Node Configuration + +On each storage node, devices are selected at `storage-node configure` time with the `--lblk` flag +and an optional device selector: + +```bash title="Configuring a storage node with all eligible block devices" +sudo {{ cliname }} storage-node configure --lblk --max-lvol +``` + +Without a selector, every eligible disk on the host is used. An eligible disk is a whole, unmounted, +unheld, and unpartitioned disk that is not the root disk (see the +[eligibility rules](../../architecture/concepts/linux-block-devices.md#device-eligibility)). + +Devices can also be selected explicitly, by name or by serial number. The three selectors are +mutually exclusive: + +```bash title="Selecting block devices by name" +sudo {{ cliname }} storage-node configure \ + --lblk --blk-names sdb,sdc --max-lvol 50 +``` + +```bash title="Selecting all eligible block devices except some" +sudo {{ cliname }} storage-node configure \ + --lblk --blk-names-exclude sda --max-lvol 50 +``` + +```bash title="Selecting block devices by serial number or WWN" +sudo {{ cliname }} storage-node configure \ + --lblk --blk-serials S3EVNX0M602707,S3EVNX0M602708 --max-lvol 50 +``` + +A requested device that is busy (mounted, held, or otherwise ineligible) is an error. The +configuration fails rather than silently skipping the device. + +The selected devices are stored in the resulting configuration file +(`/etc/simplyblock/sn_config_file`) with their name, serial, stable by-id path, size, and NUMA +assignment. As in NVMe mode, the file can be reviewed and manually edited before deployment, for +example to remove a device from the selection. + +### Partitioned Devices + +A device carrying a partition table is not eligible by default. To reuse such a device, it is marked +eligible with `--force` at configuration time: + +```bash title="Including a partitioned block device in the selection" +sudo {{ cliname }} storage-node configure \ + --lblk --blk-names sdb --force --max-lvol 50 +``` + +The wipe itself happens later, at node addition, where it has to be requested explicitly with +`--force-format`. Until then, no data is touched. + +## Storage Node Deployment and Addition + +Node deployment is unchanged: + +```bash title="Deploying the storage node" +sudo {{ cliname }} storage-node deploy --ifname eth0 +``` + +Adding the node to the cluster from a control plane node is unchanged as well, apart from one +additional flag. If partitioned devices were force-included at configuration time, with +`--force-format` the node addition is instructed to wipe partition tables and filesystem signatures +(`wipefs`) from those devices: + +```bash title="Adding the storage node while wiping partitioned devices" +sudo {{ cliname }} storage-node add-node \ + --force-format :5000 eth0 +``` + +!!! danger + `--force-format` irreversibly destroys any data on the affected devices. The device selection in + the node configuration has to be verified before the node is added. + +During node addition, each selected device is wrapped in an SPDK AIO bdev. No kernel driver unbinding +takes place, so the devices stay visible to the host OS but must not be used by anything else. The +smallest device is used as the journal device, as in NVMe mode with journal-on-device deployments. + +Everything after node addition (cluster activation, pool creation, volume provisioning, and client +connection) is identical to an NVMe-mode cluster. + +## Verification + +After activation, the devices are listed like NVMe devices, showing the device path instead of a PCIe +address: + +```bash title="Listing the storage devices of a node" +sudo {{ cliname }} storage-node list-devices +``` + +On the host, the devices remain kernel-owned and continue to be shown by `lsblk`. One `aio_` +base bdev per device is exposed by the SPDK process. diff --git a/docs/non-kubernetes/operations/lblk-device-operations.md b/docs/non-kubernetes/operations/lblk-device-operations.md new file mode 100644 index 00000000..864dfebd --- /dev/null +++ b/docs/non-kubernetes/operations/lblk-device-operations.md @@ -0,0 +1,67 @@ +--- +title: "Linux Block Device Operations" +description: "Operating lblk-mode clusters: node restarts with serial-based device resolution, adding block devices to a node, and handling failed devices." +weight: 20090 +--- + +{{ experimental }} + +Day-2 operations specific to clusters in the Linux block device mode (`lblk`) are described below. +Concepts and deployment are described under +[Linux Block Devices (lblk)](../../architecture/concepts/linux-block-devices.md) and +[Deploy with Linux Block Devices](../installation/linux-block-devices.md). + +## Node Restarts + +No `lblk`-specific handling is required for a node restart. On restart, the node's configured devices +are re-resolved serial-first against the live host inventory. Kernel device names may have changed +across a reboot (for example, `/dev/sdb` and `/dev/sdc` swapping), but the devices are still matched +correctly by their persisted serial numbers, and the AIO bdevs and the storage stack above them are +rebuilt exactly as recorded in the cluster database. + +A configured device that is missing from the host at restart is marked removed (the same semantics +as a missing NVMe controller), and the standard failed-device data migration is triggered. + +!!! info + The `--ssd-pcie` option of `storage-node restart`, by which new devices are added during a + restart, is not supported on `lblk`-mode clusters and is rejected. + +## Adding Devices to a Storage Node + +Growing a node's device set is performed by reconfiguring and re-adding the node, not at restart +time: + +1. Attach the new block devices to the host. +2. Remove the storage node from the cluster. Its data is migrated to the remaining nodes, as with any + [node replacement](replacing-storage-node.md). +3. Re-run the configuration with a selection that includes the new devices: + + ```bash title="Reconfiguring the node with an extended device selection" + sudo {{ cliname }} storage-node configure \ + --lblk --blk-names sdb,sdc,sdd --max-lvol 50 + ``` + +4. Re-add the node to the cluster. The node joins with the extended device set, and data is + redistributed onto it by the automatic rebalancing. + +Cluster capacity can alternatively be extended by +[adding a new storage node](scaling/index.md) with its own devices. + +## Failed Devices + +Device failures are handled by the same machinery as in NVMe mode. A device producing IO errors is +marked unavailable and, after the retry budget is exhausted, marked failed. The cluster map is +updated, and the affected data is rebuilt from redundancy by a data migration. + +A device whose IO hangs without erroring is caught by the `lblk` hung-IO watchdog (roughly 30 seconds +of zero progress with outstanding IO) and driven through the same unavailable, restart, and failed +path. A device that disappears from the host, through hot removal or a cloud volume detach, is +detected and treated like an NVMe hot-remove. + +A failed device is replaced by attaching a replacement device to the host and then following the +[Adding Devices](#adding-devices-to-a-storage-node) procedure. Alternatively, the whole node is +replaced, following [Replacing a Storage Node](replacing-storage-node.md). + +!!! info + SMART health information is not available for AIO-backed devices. Device health checks + (`storage-node check-device`) are limited to liveness and IO statistics.