blob: metadata-page journal with torn-write protection (md-journal v1) - #61
Open
schmidt-scaled wants to merge 10 commits into
Open
blob: metadata-page journal with torn-write protection (md-journal v1)#61schmidt-scaled wants to merge 10 commits into
schmidt-scaled wants to merge 10 commits into
Conversation
Implements the 'LVS meta-data (page) journaling with torn-write protection' specification: - per-LVS 64 MB journal ring at the highest offset of the backing virtual device (actual runtime size, not a 2 PB constant); entry = [header block: magic+crc32c+target lba][4K md page] - every md-region write (super, masks, blobstore/blob md pages) is appended to the ring FIFO and acknowledged on journal durability; the home write is deferred - in-memory copy of journaled pages + lba->slot dictionary (spinlock vs the drain thread) serves md reads as an overlay - no read amplification from the journal - async drain poller writes pages home from the in-memory copy, then zeroes both entry blocks; supersede-coalescing skips home writes covered by a newer entry; journal-full stalls appends until a slot frees - recovery on load: whole-ring scan with up to 32 parallel 64 KB reads; zeroed/corrupt entries are empty; rebuilds the contiguous valid run into buffer/dictionary/pointers and drains in background - integration via a bs_dev proxy that shrinks the visible device by the ring (no data/journal collision) with explicit shims for every vtable member; spdk_bs_init zeroes the ring in its format batch, spdk_bs_load runs recovery before the first md read Syntax-checked (gcc -fsyntax-only -Wall); not yet built or tested against a full SPDK build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tests Build the md-journal branch for real (Rocky/RHEL 9, gcc 11, --enable-debug) and bring the journal to green under a new CUnit suite. Fixes found by building and testing: - cb_args lifetime: the append pump and the drain poller passed stack-allocated spdk_bs_dev_cb_args to asynchronous base-dev IO; the bs_dev contract requires them to stay valid until completion (use-after-return crash under the UT mock dev). They are embedded in struct spdk_bs_md_journal now (single append + single drain IO in flight). - md_journal super-block feature flag (design 10): spdk_bs_load wrapped every sufficiently large device unconditionally, which would have run ring recovery - and later ring writes - over the top 64 MB of legacy (journal-less) stores. The super block now carries an md_journal flag (carved out of reserved[], set once at format time); spdk_bs_load pre-reads page 0 raw and wraps the device only when a valid signature carries the flag, otherwise the legacy path is byte-for-byte unchanged. A flagged store whose device cannot carry the ring fails the load with -EILSEQ instead of loading unprotected. - spdk_bs_init now arms write/read interception only after the super block (with the flag) is home-durable (bs_init_persist_super_cpl): format writes go raw, so any store whose init was acknowledged always carries the flag on disk; md writes after init completion are journaled as before. The flag never changes over a store's lifetime, so even a stale or torn-on-drain raw super still routes the load through recovery (I1 holds across power-off). New unit-test suite test/unit/lib/blob/blob_md_journal.c (cases U1-U12 of md_journal_test_plan.md) driving the journal through its proxy bs_dev against a manual-completion mock dev: append/ack ordering (I1), read overlay, drain + entry zeroing (I2), supersede-coalescing, ring full stall (8191 + guard), recovery (basic, torn tail entry, duplicate LBAs), power-off snapshot simulation, proxy geometry/passthrough, multi-page FIFO append (I3), destroy with in-flight IO. blob_ut now compiles the journal in (blobstore.c includes it); suite wired into test/unit Makefiles and unittest.sh. Results (c6i.4xlarge, Rocky/RHEL 9): blob_md_journal_ut 12/12 passed (117837 asserts, valgrind clean), blob_ut 376/376 passed, blob_bdev_ut 8/8, lvol_ut 37/37, vbdev_lvol_ut 23/23; full tree make -j16 clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single-node integration testing (real spdk_tgt, lvstore on a 4 GiB file-backed AIO bdev, journal active) hit sporadic blob-md CRC mismatches on lvol create. Root cause is a read-vs-drain race in the overlay path: the proxy issued the home read first and looked up the lba->slot dictionary only on read completion. The drain poller could complete the home write, zero the ring entry and drop the dictionary entry while that read was in flight - the completion-time lookup then missed and the caller was served the pre-drain (stale) page content the concurrent read had captured. The design's assumption that a completion-time lookup is safe was wrong. Fix: snapshot the dictionary hits (page copies, under the lock) at read ISSUE time and overlay the snapshots on completion. Any version between issue and completion is a linearizable read result, and a miss at issue time implies the home write was already durable before the read was issued. A failed overlay allocation now completes the read with -ENOMEM instead of falling back to a raw read that could serve pages the journal has not written home yet. Design doc section 7 updated accordingly. Unit tests: the mock dev now snapshots read data at issue time (worst-case device reordering of concurrent IOs) and new case U13 reproduces the race deterministically: append acked, home read issued, drain overtakes (home write + zero + dict drop), read completes with stale device data -> the issue-time snapshot must win. 13/13 pass; blob_ut still 376/376. New test/md_journal/mdj_integration.py (cases I1-I5 of the plan, ultra-independent): lvol/snapshot lifecycle under active journal, 4x kill -9 rounds mid create/delete workload with RPC-ack tracking (every acked object present after recovery, every acked delete stays deleted, no torn-md load abort), torn-write injection into ring slots while down, 480 parallel create/delete pairs, and drain-to-zero verification on the live ring. Two consecutive full-suite passes on the build host; 'md journal enabled' confirmed on every load; zero md errors in the target log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Torn-write states cannot occur naturally on this test rig: kill -9 never tears a write (io_submit'ed IOs complete in-kernel, unsubmitted ones never start) and AWS Nitro EBS/instance store adds 16 KiB torn-write prevention on top. All torn states are therefore synthesized by injection, modeling the 512B-sector-atomic worst case of the distr virtual device the design targets. Doing that exposed a load-order gap: spdk_bs_load read the super block BEFORE bs_md_journal_enable armed interception (the md limit is only known after parsing that very super block), so the super always came raw from its home LBA. After a crash the newest super may still sit in the ring - the home copy stale, or torn mid-drain by the power loss. A torn home super aborted the load (crc failure) although a valid acked copy sat in the journal; a stale one would silently roll back acked metadata state. Fix: recovery completion arms interception for the WHOLE proxy range (a dictionary miss passes through untouched, so overlaying any LBA is correct); bs_parse_super then tightens the limit to the md region as before. Fresh format is unchanged (interception stays off until the super with the feature flag is home-durable). Tests: - unit: recovery path now asserts the wide interception window; 13/13 pass, blob_ut 376/376. - integration I6 (new): synthesize bit-exact valid ring entries (magic + raw CRC32C) for three blob md pages and the super block, tear all four home pages on the raw file (512B-granular, sector 0 of the super kept for the load probe), restart: the store must load and repair every page byte-exactly from the ring. Passes, along with two consecutive full I1-I6 suite runs; test plan updated with the representativeness note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The simplyblock fork (nvmf-blocking-from-base) extends every spdk_bs_dev IO op with a trailing struct spdk_bs_io_opts * carrying priority class, geometry id and the special-io flag, and callees dereference it unconditionally. Ported after rebasing md-journal onto that branch: - proxy ops forward the caller's bs_io_opts unchanged (md overlay reads included), so passthrough IO keeps its routing bits; - journaled md writes snapshot the caller's opts per ring entry and the entry header now persists them (priority/geometry/special_io), so the deferred home write - including one replayed by recovery after a crash - carries the same routing as the original write; - journal-internal ring IO (entry writes, recovery reads, zero-on-drain) uses default zeroed opts, matching bs-level md sequences in request.c; - spdk_bs_load's raw super-block probe passes default opts; - unit-test mock dev and all proxy call sites updated accordingly. On-disk note: the entry header grows by 4 bytes (within the reserved 4K header block); pre-port rings are not format-compatible, which is fine pre-release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The journal append write passed a stack-allocated iovec[2] to base->writev. bs_dev consumers may read the iov array long after submission returns: the distrib bdev queues the request and copies the payload on its own poller thread, so the dead stack frame turned into a wild memmove (SIGSEGV in alg_io_split copy_to_buf during bdev_lvol_create_lvstore on the raid/distrib stack). aio- and malloc-backed devices consume iovs at submit time, which is why the spdk-only phase-1 runs never tripped this. Move the iov pair into spdk_bs_md_journal — there is exactly one append IO in flight by design (I3), like append_cb_args. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
blob_persist_zero_pages zeroes released md pages (blob delete zeroes the blob's first page; every persist zeroes the released chain pages) with write_zeroes on md-region LBAs. The proxy passed those raw to the base dev, breaking FIFO ordering with journaled writes of the same pages: an older journaled copy still waiting in the ring would later be drained home OVER the zeroes, resurrecting the page. Observed on the ultra distr stack as every acked-deleted lvol reappearing after kill -9 + reload (mdj integration test, kill round 1). Route md-region write_zeroes and unmap (page-aligned; unmapped ranges read back as zeros, so an all-zero page is equivalent) through the append queue as payload-less all-zero page entries. Non-md ranges and unaligned ranges still pass through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recovery ran only in spdk_bs_load, but the product never loads the lvstore on failover: the secondary loaded it when the LVS was created or activated, and takeover is bdev_lvol_update_lvstore + bdev_lvol_set_leader_all (or the IO-driven spdk_bs_update_on_failover). The ring is shared state on the shared device while the buffer and dictionary that overlay md reads from it are per process, so a promoted peer served home pages for everything the dead leader had acknowledged but not drained, and then appended at its own stale head over those entries, breaking the contiguous run recovery derives tail/head from. spdk_bs_update_live now re-runs recovery (bs_md_journal_rescan) before it re-reads the super block whenever the whole store is reloaded (id == 0), which covers both promotion paths; the per-blob variant used by bdev_lvol_register keeps its old cost. The rescan waits for the append/drain pipeline to quiesce, drops the dictionary and both pointer pairs, re-reads the ring as recovery does and re-arms the drain poller, so the new leader inherits and drains the dead leader's backlog. Also adds the ring introspection and drain test hook the failover suite needs, bdev_lvol_get_md_journal_stats and bdev_lvol_set_md_journal_drain: the drain keeps the ring at about one entry under any md rate the blobstore can produce (six parallel md clients still left zero entries at kill time), so neither recovery-with-entries nor journal-full is reachable by workload alone. Verified on the two-instance failover rig (ultra mdj_failover_tests.py): promotion of a preloaded peer now logs the rescan followed by "md journal recovery: N entries", and a 915-entry backlog left by a kill -9 is replayed entry for entry and drained to empty. Design doc: new section 8.1 for the rescan; section 11.2 records that single-writer fencing does NOT exist in the data plane (test F3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drain poller is a background writer the pre-journal md path did not have: it keeps pushing this process's in-memory md pages to their home LBAs on the SHARED device with no IO to trigger it. Nothing stopped it when the node lost leadership, so a demoted but still living node kept writing metadata behind the new leader's back. That is not hypothetical. The product fences by fail-stop - a writer conflict is detected by the JM/JC layer, which signals the offending node to freeze and cancel IO and then abort - but the network-outage path (spdk_lvs_change_leader_state, groupid 0) demotes the LVS and leaves the process running, and a promotion elsewhere can precede the abort in the conflict case. In both windows the old leader still reaches the shared device. spdk_bs_set_leader() now tells the journal, and the drain poller idles while the node is not the leader; the buffer it was holding is rebuilt from the ring by bs_md_journal_rescan() if this node is promoted again. Together with that rescan the lifecycle is symmetric: demote stops writing, promote re-reads the ring before it resumes. The new state is reported by bdev_lvol_get_md_journal_stats as drain_demoted so the failover suite can assert on it (test F3b). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drain no longer runs on a demoted node, but that node can still append: the lvol layer gates destroy and async delete on lvs->leader, not create, and it deliberately allows a sync delete on a non-leader - so "a non-leader never writes md" is not the fork's model today. Entries such a node appends stay in its ring until it is promoted and rescans, or the real leader's head marches over those slots. Deciding this belongs to the lvol layer (refuse md-mutating work on a non-leader) or to an owner/epoch in the ring; documented rather than changed unilaterally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blobstore metadata-page journal (md-journal v1)
Adds a write-ahead journal for blobstore metadata pages, closing the torn-write inventory found in the R26.2-PRE audit (root/chain/extent/used_blobids pages written in place, silent used_blobids loss, super block without backup): every md-page write is first appended to a dedicated journal region with per-record CRC framing, so a torn in-place write is always repairable from the journal on the next load. Design doc:
blobstore_metadata_journal_design.md.Commits (bottom-up)
cb_argslifetime fix, unit tests.bs_devio-opts vtable.write_zeroes/unmapas zero-page appends, so metadata trims are replayable like writes.Validation
ultra:md-journal-latest+ sbclimd-journal-05ed69d6): 60/60 passed, zero fio faults, zero cluster suspensions, ~120 node recoveries clean.🤖 Generated with Claude Code