Add lblk cluster mode (Linux block devices via SPDK AIO bdevs) + campaign fixes - #1224
Open
schmidt-scaled wants to merge 4 commits into
Open
Add lblk cluster mode (Linux block devices via SPDK AIO bdevs) + campaign fixes#1224schmidt-scaled wants to merge 4 commits into
schmidt-scaled wants to merge 4 commits into
Conversation
New cluster-global device mode chosen at cluster create (--device-mode
nvme|lblk, deploy-time only). In lblk mode, eligible Linux block devices
(unmounted, unheld, unpartitioned whole disks; partitioned only with
--force-format on add-node, which wipes them host-side) are wrapped in
one SPDK AIO bdev per device. Everything from alceml upward is unchanged;
nvme-tcp/rdma fabric is untouched.
- Selection at `sn configure --lblk` (and the k8s node_configure twin) by
device name include/exclude or serial number; the node config file
carries an editable `lblk_devices` list ({name, serial, by_id, size,
numa}) parallel to ssd_pcis, validated as exactly-one-device-source.
- Identity is SERIAL-FIRST (lsblk SERIAL -> WWN -> stable synthetic id):
add-node persists the selection on the node record; restart re-resolves
serial -> current kernel name (stored name as fallback), so device
renames across reboots cannot attach the wrong disk. AIO bdev names are
derived from the serial (collision-safe) and stable across restarts.
The serial-keyed restart reconcile works unchanged; missing device ->
STATUS_REMOVED, new -> STATUS_NEW, same as nvme.
- SPDK launch in lblk mode never passes an empty PCI allowlist (DPDK
treats empty as allow-all; the k8s path passes PCI_ALLOWED="" today) —
a host-bridge placeholder 0000:00:00.0 is used instead, and no
vfio/uio binds ever happen, so SPDK cannot claim kernel disks.
- Failure parity (control plane only): the distrib error_* event path is
already bdev-generic; the hung-IO gap (AIO has no bdev_nvme timeout_us/
action_on_timeout) is closed by a device_monitor watchdog using
queue-depth-sampled iostat — inflight IO with zero completion progress
across 3 polls (30s) feeds io_error + UNAVAILABLE with a countable
LOCAL_FAILURE cause into the existing flap/auto-restart/FAILED/
migration machinery; >=2 simultaneously stalled devices escalate to a
node auto-restart; RPC failures freeze (never advance) the counters.
Device disappearance from the host inventory drives device_remove (the
SPDK_BDEV_EVENT_REMOVE treatment) after a 2-poll debounce. The
late-event gate, reset (liveness probe — never delete/recreate the aio
bdev in place), SMART info, restart_device and new_device_from_failed
are mode-aware.
- New snode endpoints: GET /blockdevices (whole-disk inventory with
eligibility fields, by-id path, NUMA) and POST /wipe_block_device
(re-validates busy state, wipefs partitions-then-disk), on both docker
and k8s agents.
- Phase-1 scope: lblk requires journal-on-device (GPT-partition JM mode
is nvme-only); --ssd-pcie/--reattach-volume are rejected on lblk nodes.
tests: 93 unit tests (eligibility/onboarding/watchdog/device_controller)
+ 10 FDB-backed integration tests (model round-trips, restart identity
contract over renamed devices, watchdog -> real state machine, flap-limit
force-FAILED + migration, disappearance -> device_remove, reset). Full
unit tier 1005 green, ruff clean.
External follow-ups: validate run_distr_with_ssd.sh tolerates the
placeholder -A on a test node; confirm the fork's bdev_get_iostat carries
qd-sampling fields; alceml over 512e aio bdevs (block_size currently
omitted, 4096 fallback if needed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first AWS lblk deploy (2x50G EBS per 32 GiB host) failed every `sn configure --lblk` with "Free memory ... less than required 127228418457": node_config_min_sys_memory charged 2 GiB + the FULL device capacity. The nvme path nominally does the same but always measures zero — capacity is read via `nvme list` after the devices were unbound from the kernel driver — so the de-facto contract (and the documented intent, "plus 0.2% of the storage") is a small fraction. Apply the documented 0.2% factor for lblk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gate Two control-plane gaps exposed by the 2026-08-05 lblk soak bring-up (both generic, neither lblk-specific): 1. Zombie SPDK on failure cleanup. When add-node/restart aborts with "node did not come up", _kill_spdk_until_dead verified death via spdk_process_is_up — an RPC-Unix-socket probe that false-negatives an SPDK which booted but never brought its RPC up. Combined with spdk_process_kill's deliberately detached container remove (fast peer termination) losing the race against the container restart policy, the "confirmed down" SPDK survived, squatting ~all hugepages and starving every subsequent add-node retry on the host. New agent endpoint spdk_process_cleanup: clears the restart policy, removes synchronously, and reports success only when the containers are verifiably GONE (k8s agent: alias of its already-synchronous pod-delete-and-poll). _kill_spdk_until_dead prefers it and falls back to the legacy kill + socket-poll for older agents. 2. JM-mesh activation gate. Nodes that joined through add-node retries ended with peers missing their remote_jm_* controllers; the cluster activated and reported healthy while a third of the journal mesh was unreachable. First journal load excluded those JMs, n_safe_jms collapsed and JCERR cascaded cluster-wide. New storage_node_ops.verify_jm_mesh_coverage(): every ONLINE node must hold live remote bdevs for the remote JMs it references, with a one-shot _connect_to_remote_jm_devs repair. Wired into _cluster_activate: FRESH activation fails on unrepaired holes; RE-ACTIVATION is a recovery path that may legitimately run with one or two nodes unhealthy — the verifier skips JMs whose owner is not ONLINE and the gate only warns. tests: tests/unit/test_jm_mesh_and_spdk_cleanup.py (11 cases); full unit tier 1000 green, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
md-journal test campaigns now run against the fork's md-journal branch (blobstore metadata journal work) instead of ultra main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| def test_reappearance_clears_counter(self): | ||
| dev = _aio_dev(serial="S1") | ||
| gone = self._node_with_inventory([dev], []) |
| try: | ||
| rpc_client.bdev_set_qd_sampling_period( | ||
| dev.nvme_bdev, constants.AIO_QD_SAMPLING_PERIOD_US) | ||
| except Exception: |
| container.id[:12], exc) | ||
| try: | ||
| container.remove(force=True) | ||
| except NotFound: |
| client.containers.get(name) | ||
| ok = False | ||
| logger.error("cleanup: %s still present after remove", name) | ||
| except NotFound: |
| base = f"/sys/block/{name}" | ||
| try: | ||
| holders.extend(os.listdir(f"{base}/holders")) | ||
| except OSError: |
| if entry.startswith(name): | ||
| try: | ||
| holders.extend(os.listdir(f"{base}/{entry}/holders")) | ||
| except OSError: |
| holders.extend(os.listdir(f"{base}/{entry}/holders")) | ||
| except OSError: | ||
| pass | ||
| except OSError: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lblk cluster mode: Linux block devices as storage via SPDK AIO bdevs
Rebased onto current main (152613f); the restart-claim fix formerly carried on this branch is already on main (17ebddb) and dropped out of the diff. Four commits:
1. lblk cluster mode (7757cd2)
Cluster-global
device_mode: nvme|lblkchosen atcluster create. Inlblkmode, eligible Linux block devices (whole, unmounted, unheld, unpartitioned disks — partitioned only with--force/--force-format) are wrapped in SPDK AIO bdevs; everything from alceml upward is unchanged, and the inter-node fabric (nvme-tcp/rdma) is untouched.sn configure --lblkby--blk-names/--blk-names-exclude/--blk-serials; config file manually editable.aio_<sanitized_serial>is stable.bdev_nvmeIO timeout) is closed by a hung-IO watchdog in DeviceMonitor (qd-sampling + iostat progress, 3×10 s → UNAVAILABLE; ≥2 stalled devices → node auto-restart); device disappearance handled via /blockdevices inventory sweeps; late-event gate, reset, restart, and new-device-from-failed all have mode-aware branches.0000:00:00.0(never an empty list — empty means DPDK allow-all on k8s).num_partitions_per_dev == 0);--ssd-pcierestart-time growth is rejected on lblk clusters.2. sys-memory sizing fix (5541736)
sn configure --lblkdemanded 2 GiB + full device capacity (102 GiB for a 100 GiB node). Now 2 GiB + 0.2 % of capacity. The nvme path accidentally measured zero (capacity read after driver unbind) and is unified on the same formula.3. Zombie-SPDK cleanup + JM-mesh activation gate (6e7fcb2)
Two field findings from the lblk soak campaigns, both mode-independent CP bugs:
spdk_process_cleanup(clears restart policy, synchronous verified remove);_kill_spdk_until_deadprefers it with legacy fallback.remote_jm_*controllers, silently degrading JC quartets until the first journal load collapsed them. Newverify_jm_mesh_coverage+ activation gate: fresh activation hard-fails on unrepaired mesh holes; re-activation only warns (recovery with unhealthy nodes must never be blocked). 11 unit tests.4. env_var: SPDK fork image →
ultra:md-journal-latest(8385dd9)Points the default fork image at the md-journal branch (blobstore metadata journal, simplyblock/spdk#61). Flagged for reviewer decision: merging this changes the default SPDK image for every deployment from this branch — drop this commit if main should stay on
ultra:main-latestuntil spdk#61 lands.Validation
constantsimportdevice_controller.pynow needs — fixed in the feature commit).ultra:md-journal-latest: full 60-scenario dual-node-outage churn soak (graceful / forced / container_kill / host_reboot / 65 s network partitions × unrelated / P-T / P-S pairs) — 60/60 passed, zero fio faults, zero cluster suspensions.🤖 Generated with Claude Code