diff --git a/blobstore_metadata_journal_design.md b/blobstore_metadata_journal_design.md new file mode 100644 index 00000000000..31786ceb1f0 --- /dev/null +++ b/blobstore_metadata_journal_design.md @@ -0,0 +1,247 @@ +# LVS metadata (page) journaling with torn-write protection — design + +Status: rev 3 — aligned precisely to the specification +"LVS meta-data (page) journaling with torn-write protection" (2026-07-29). +Scope: simplyblock SPDK fork, branch `md-journal`, `lib/blob`. + +## 1. Problem + +The lvstore reports a **4K logical block** and runs the blobstore on a **4K page**. Blobstore +crash-consistency assumes a single 4K metadata-page write is **atomic**. The backing store +guarantees atomicity only at the **512-byte sector**, so a 4K metadata write can **tear** on power +loss. Only blobstore **metadata** must be made torn-safe. + +Consequences today (confirmed by source mapping): +- Torn **root/extent page** → page CRC fails → `bs_load_iter` aborts the **entire lvstore load**. +- Torn **`used_blobids` mask page** → no CRC, seeds dirty-recovery replay → **silent loss of a + blob**. + +## 2. What is journaled + +Per the spec: **all md updates** — super block, bitmap (mask) pages, blobstore md pages, per-blob +md pages. Concretely every 4K write whose target falls inside the blobstore metadata region: + +| Structure | Journaled | +|---|---| +| Super block (page 0) | yes | +| `used_pages` / `used_clusters` / `used_blobids` mask pages | yes | +| Blob root pages, COW chain pages, extent pages (the md-page region) | yes | +| Data cluster writes | no (pass through) | + +## 3. Journal placement and format + +- Per LVS, a **64 MB region at the highest offset of the virtual underlying space**: + `journal_start = dev_size_bytes − 64 MB`, where `dev_size_bytes` is taken from the **actual + device size at runtime** (`blockcnt × blocklen`), never a hard-coded 2 PB. +- The region is a **ring buffer** of fixed-size entries. +- **Entry = 2 blocks (8 KB total)**: + - **block 0 — page header**: `{ u32 checksum (CRC32C of the md page), u64 target_lba }`, + rest of the block zero. The checksum comes **with the write from the caller** (the caller + hands us the finished md block[s] including checksum[s]). + - **block 1 — the 4K md page**, byte-exact home content. +- A zeroed or checksum-mismatching entry is **empty** by definition (torn journal writes are + self-detecting; they were never acknowledged, so discarding them is always correct). +- 64 MB / 8 KB = 8192 entries. One entry slot is kept as a **guard** (ring is "full" at 8191 + used) so a completely-valid full ring cannot become ambiguous on recovery. + +## 4. In-memory state (read-amplification avoidance) + +To avoid read amplification, journaled pages are **never read back from the journal in normal +operation**: + +- After a journal write completes, the md page is **copied into an in-memory buffer** (so it + survives release of the caller's write buffer on IO completion). +- An **in-memory dictionary maps target LBA → offset of the newest copy in that buffer/SGL**. +- **Journal head and tail pointers are kept in memory as two pairs**: one pair for the in-memory + buffer, one pair mirroring the on-disk ring state. Nothing is persisted; recovery rebuilds both + pairs from the ring content (§7). +- The dictionary (and buffer slot reuse) is protected by a **spinlock**, because md reads race + with the LVS drain thread. + +## 5. Write path + +On every md-page write (per 4K page, in issue order): + +1. Build the header block (checksum from caller, target LBA) and write + `[header][md page]` to the ring slot at the in-memory head; advance head. +2. On journal-write completion: copy the page into the in-memory buffer, point + `dict[target_lba]` at it (newest wins), advance the on-disk head mirror, and + **complete the IO back to the caller**. The home write has not happened yet and the caller + must not care. +3. **Journal full** (head would run into tail's guard slot): the write **waits** until the LVS + drain thread frees a slot, then proceeds. No error, no bypass. + +Multi-page md writes (mask flushes, chain batches) are appended page-by-page FIFO and completed +when all their entries are durable. + +## 6. Drain (LVS asynchronous thread) + +A background poller on the blobstore md thread loops comparing head and tail. While they differ, +for the entry at tail: + +1. Take the target LBA from the entry (first block) and write the md page (second block — + served from the in-memory copy, not a journal read) to its home LBA. +2. On home-write completion, **unmap (zero) both journal blocks** of the entry. +3. Advance tail (both pairs); drop `dict[target_lba]` **iff** it still points at this entry's + buffer slot (a newer copy may exist further up the ring), then release the buffer slot. + +Ordering invariant: a slot is only reused after its home write is durable **and** its journal +blocks are zeroed — this is what keeps "non-empty ⇒ not yet applied" true for recovery, and the +zeroing is what makes the valid region of the ring contiguous. + +## 7. Read path + +On every md read: **first look up the dictionary** (under the spinlock). For each 4K page of the +read range that hits, serve it from the in-memory copy (validate its checksum); pages that miss +are read from home. Implementation: **snapshot the dictionary hits (page copies) at read issue +time**, issue the home read, and overlay the snapshots on completion. The snapshot at issue is +load-bearing: the drain can complete a home write, zero the entry and drop the dictionary entry +*while the home read is in flight*, and the concurrent home read may still return the pre-drain +page — a completion-time lookup would then miss and serve that stale data (found as a blob-md +CRC mismatch in single-node integration testing). + +## 8. Sudden power-off and fail-over (recovery) + +Run by whichever node next loads the LVS (secondary/tertiary on failover, same node after +power-off), **before any other md read**: + +1. **Read the whole 64 MB journal** using up to **32 parallel 64 KB IOs**. +2. Validate every entry: an entry is valid iff it is not all-zero and the header checksum + matches the payload page. **Zeroed and corrupted entries are both treated as empty** (torn + writes inside the journal itself are expected and harmless — they were never acknowledged). +3. The valid entries form one contiguous run in ring order (guaranteed by §6's zero-on-drain and + the §3 guard slot): its ends are the recovered **tail and head**; set both in-memory pairs + accordingly. +4. Fill the **in-memory buffer and dictionary** from the valid entries in ring (FIFO) order — + for duplicate LBAs the later entry wins. +5. Continue normal load: md reads are already correct via §7, and the **LVS drain thread works + the backlog off in the background** — no upfront redo pass, no load stall. +6. Recovery completion arms read/write interception for the **whole device range** immediately: + the super block is read before the metadata layout is known, and after a crash its newest + version may still sit in the ring (home copy stale or torn by the power loss mid-drain). + Overlaying is correct for any LBA (a dictionary miss passes through untouched); the range is + tightened to the metadata region once the super block is parsed. + +### 8.1 Promotion of an already-loaded peer (rescan) + +The product does **not** load the LVS at failover. The secondary/tertiary loaded it when the +LVS was created or activated, and takeover is `bdev_lvol_update_lvstore` + +`bdev_lvol_set_leader_all` (`storage_node_ops` leaderless recovery / failover), plus the +IO-driven reactive promotion (`spdk_bs_update_on_failover`). Neither loads the blobstore, so +§8 recovery would never run on the node that becomes the new leader. + +That is not survivable: the ring is shared state on the shared device, the buffer/dictionary +that overlays reads from it is **per process**. A peer's ring view is a snapshot as of its own +load; every entry the leader appended (and acknowledged) afterwards is invisible to it. A peer +promoted without a rescan therefore + +- serves the **home** page for every page the dead leader acknowledged but did not drain — the + acknowledged md is lost, which is exactly what the journal exists to prevent; and +- appends at **its own stale head**, overwriting those undrained entries and breaking the single + contiguous run that §8 step 3 relies on, so even a later crash-recovery cannot get them back. + +`spdk_bs_update_live()` therefore re-runs recovery (`bs_md_journal_rescan()`) before it re-reads +the super block whenever the whole store is reloaded (`id == 0`), which covers the explicit +promotion RPC and the reactive failover path. The rescan waits for the append/drain pipeline to +quiesce, drops the dictionary and both pointer pairs, re-reads the ring exactly as §8 does, and +re-arms the drain poller, so the new leader inherits the dead leader's backlog and drains it. +The per-blob variant of the same call (`bdev_lvol_register`, `id != 0`) does not rescan: it runs +per lvol create in a live cluster and the cost would land on the create path. + +Measured on the two-instance failover rig (ultra `mdj_failover_tests.py`, phase 3): without the +rescan a promotion logged no recovery at all; with it, promotion logs +`md journal rescan: re-reading the ring on takeover` followed by the normal +`md journal recovery: N entries` line. + +## 9. Why this is correct (invariants) + +- **I1 — ack after journal durability**: the caller sees completion only once the entry is on + disk. Hence any torn/invalid entry found by recovery was never acknowledged → safe to drop. +- **I2 — reuse after home durability + zeroing**: a slot is recycled only after its page is home + and the entry is zeroed → every acknowledged-but-not-yet-home page is present and valid in the + ring; recovery cannot miss one. +- **I3 — FIFO order end to end**: append order = issue order; drain and recovery process in ring + order; duplicates resolve newest-wins. The blobstore's own prefix-consistent write ordering is + therefore preserved; no transactions or multi-page atomicity are needed. +- The journal never relies on any write being atomic — only on **torn-write detection** (the + per-entry checksum). + +## 10. Integration points (fork) + +- New module `lib/blob/blob_md_journal.[ch]`: ring + buffer + dictionary + drain poller + + recovery; owned by `struct spdk_blob_store`. +- The journal claims the top 64 MB by presenting the blobstore a **shrunk device** + (`blockcnt − 64 MB worth`), so data clusters can never collide with the ring; the journal + itself addresses the base device raw. +- **Write interposition** at the md write-issue points / md LBA range; data-path writes are + untouched. +- **Read interposition** as §7 overlay. +- `spdk_bs_init`: zero the journal region when formatting. +- `spdk_bs_load`: run §8 recovery first. +- Feature/format flag in the super block: journaling active only for stores formatted with the + reserved region (legacy stores load unchanged). + +## 11. Risks / validate first + +1. **Completion ⇒ durability** on the backing dev (distr/JC committed) — required for I1/I2; + today's md writes already assume this. +2. **Single-writer fencing** — one appender at a time; re-drain is idempotent, concurrent append + is not. + + **How the product actually fences: fail-stop.** Leadership moves only when the old leader is + gone — its SPDK process died (abort, segfault, container kill, host reboot) or, on a network + outage, the node aborts itself from inside. There is no scenario in the intended design where + a healthy old leader keeps serving while a peer is promoted, and the journal inherits that + guarantee rather than adding one. + + **What phase-3 test F3 establishes (2026-08-05).** The journal contributes *no* fence of its + own, so the assumption above carries all the weight. Freezing a leader with SIGSTOP, promoting + the peer and thawing the old leader — deliberately breaking fail-stop — it accepted every + metadata operation: `bdev_lvol_get_lvstores` still reported `"lvs leadership": true`, three + creates and a sync delete were acknowledged, its ring head advanced 2673 → 2688 (15 entries) + while the new leader was at head 23, and its drain poller wrote those pages to their home LBAs + on the shared device. Nothing logged a leadership rejection. The blob layer's `is_leader` + checks cover only async delete and cleanup (`bs_delete_blob_non_leader`, + `blob_clear_clusters_async`, `bs_cleanup_*`); the lvol layer gates on `lvs->leader`, which a + stale node still has set. Nothing consults the device: with no epoch in the super block or the + entry header, a second appender is indistinguishable from the first. + + **Why this matters even under fail-stop: the journal widens the blast radius.** Before the + journal a stale writer wrote stale md pages to their home LBAs — damaging but page-local. + Now it also mutates *shared ring structure* with pointers that have diverged from the new + leader's (2673 vs 23 above): it appends into slots the new leader believes are free, and its + drain zeroes slots and writes home pages the new leader's recovery depends on, so I2 (a slot + is recycled only after its page is home) and I3 (FIFO, one contiguous run) can both break and + recovery can mis-derive tail/head. Any window where fail-stop is soft is therefore more + expensive than it used to be — and such windows are real, not hypothetical: the self-abort on + a network outage is a timed reaction and **a promotion elsewhere can precede it** (confirmed + with the product owner, 2026-08-05), and a reactor stalled by host swap thrash that later + resumes (MCD incident 2026-07-13) looks exactly like the SIGSTOP above. The exposure is + bounded by how long the old leader can still reach the shared device after the peer is + promoted. + + **Partially addressed:** `spdk_bs_set_leader()` now stops the drain poller when the node is + not the leader (`bs_md_journal_set_leader`), because the drain is a *background* writer that + the pre-journal md path did not have — it pushed this node's pages to the shared device with + no IO to trigger it. A demoted-but-living node (network-outage path, or the window before a + conflict abort completes) therefore no longer writes metadata home behind the new leader. + Test F3b confirms `drain_demoted` goes true on demotion and the held entries stop moving. + + **Still open after that:** a demoted node can still *append*. The lvol layer gates destroy and + async delete on `lvs->leader` but not create, and it deliberately permits a **sync delete on a + non-leader** — so "a non-leader never writes md" is not the fork's model today. Entries such a + node appends now stay in its ring until it is promoted (and rescans) or the real leader's head + marches over those slots. Deciding this needs the lvol layer: either md-mutating work is + refused on a non-leader, or the ring gets an owner/epoch. + + **Not implemented, and deliberately so:** closing the general case needs a fence the device can see — a + monotonically increasing leadership epoch in the super block, carried in every entry header, + with appends refused once the on-disk epoch has moved on. That is a change beyond the journal + (the epoch has to be owned by whoever grants leadership) and is out of scope here; it is + recorded so the trade-off is a decision rather than an oversight. +3. **Journal-full behavior** under md-heavy bursts (mass create/delete): writers stall until the + drain frees slots — benchmark; drain batches multiple entries per poll. +4. **Unmap-vs-zero**: if the device's unmap does not guarantee deterministic zero-read, use + explicit write-zeroes for the two entry blocks. +5. **Migration**: legacy stores without the reserved region keep the old (unprotected) path + until reformatted/migrated. diff --git a/include/spdk/blob.h b/include/spdk/blob.h index 255bfcb7e5f..7b4a5fd237d 100644 --- a/include/spdk/blob.h +++ b/include/spdk/blob.h @@ -480,6 +480,28 @@ void spdk_bs_set_role(struct spdk_blob_store *bs, node_role_t role); node_role_t node_role_from_string(const char *str); const char *node_role_to_string(node_role_t role); void spdk_bs_set_read_only(struct spdk_blob_store *bs, bool state); + +/* Metadata-journal introspection and test control (blob_md_journal.h). + * The stats are a snapshot of the ring as this process sees it. Pausing the + * drain is a test hook: it lets a workload build up a backlog of + * acknowledged-but-not-home pages, which is otherwise almost impossible to + * observe because the drain keeps up with any md rate the blobstore can + * produce. */ +struct spdk_bs_md_journal_stats { + bool enabled; /* journal present and intercepting */ + bool drain_paused; + bool drain_demoted; /* drain stopped: not the leader */ + uint32_t num_slots; + uint32_t used_slots; + uint32_t mem_head; + uint32_t mem_tail; + uint32_t disk_head; + uint32_t disk_tail; +}; + +int spdk_bs_get_md_journal_stats(struct spdk_blob_store *bs, + struct spdk_bs_md_journal_stats *stats); +int spdk_bs_set_md_journal_drain_paused(struct spdk_blob_store *bs, bool paused); void prepare_s3_clusters(struct spdk_blob* blob, uint64_t *clusters, uint32_t num_clusters); bool spdk_blob_get_offset_allocate(struct spdk_blob *blob, uint64_t offset); bool spdk_blob_check_offset_valid(struct spdk_blob *blob, uint64_t offset, uint64_t length); diff --git a/lib/blob/Makefile b/lib/blob/Makefile index 719f368241e..3f51c3ca537 100644 --- a/lib/blob/Makefile +++ b/lib/blob/Makefile @@ -9,7 +9,7 @@ include $(SPDK_ROOT_DIR)/mk/spdk.common.mk SO_VER := 12 SO_MINOR := 0 -C_SRCS = blobstore.c request.c zeroes.c blob_bs_dev.c +C_SRCS = blobstore.c request.c zeroes.c blob_bs_dev.c blob_md_journal.c LIBNAME = blob SPDK_MAP_FILE = $(abspath $(CURDIR)/spdk_blob.map) diff --git a/lib/blob/blob_md_journal.c b/lib/blob/blob_md_journal.c new file mode 100644 index 00000000000..0d6d6979128 --- /dev/null +++ b/lib/blob/blob_md_journal.c @@ -0,0 +1,1422 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (C) 2026 Simplyblock GmbH. + * + * LVS metadata (page) journaling with torn-write protection. + * See blob_md_journal.h and blobstore_metadata_journal_design.md. + */ + +#include "spdk/stdinc.h" +#include "spdk/blob.h" +#include "spdk/crc32.h" +#include "spdk/env.h" +#include "spdk/queue.h" +#include "spdk/thread.h" +#include "spdk/util.h" +#include "spdk/log.h" + +#include "blob_md_journal.h" + +SPDK_LOG_REGISTER_COMPONENT(blob_md_journal) + +#define JOURNAL_SLOT_INVALID UINT32_MAX +#define DICT_EMPTY_KEY UINT64_MAX +#define DICT_TOMBSTONE_KEY (UINT64_MAX - 1) +/* twice the slot count, power of two -> low load factor, linear probing */ +#define DICT_NUM_BUCKETS (BS_MD_JOURNAL_NUM_SLOTS * 2) + +struct md_journal_entry_hdr { + uint32_t magic; + uint32_t crc; /* CRC32C of the 4K md page */ + uint64_t target_lba; /* home LBA, base-dev blocks */ + /* bs_io_opts of the originating md write (simplyblock fork routing + * hints); persisted so the deferred home write — including one issued + * by recovery after a crash — replays with identical routing */ + uint8_t io_priority; + uint8_t io_geometry; + uint8_t io_special; + uint8_t io_rsvd; +}; + +/* journal-generated IO (ring writes/reads, entry zeroing) uses default + * routing, matching bs-level md sequences (request.c: geometry 0) */ +static struct spdk_bs_io_opts g_ring_io_opts; + +struct md_journal_dict_bucket { + uint64_t lba; /* DICT_EMPTY_KEY / DICT_TOMBSTONE_KEY / target lba */ + uint32_t slot; +}; + +struct md_journal_append_op { + /* caller payload; exactly one of payload / iov is set */ + void *payload; + struct iovec *iov; + int iovcnt; + uint64_t lba; /* first target lba */ + uint32_t lba_count; /* total blocks */ + uint32_t blocks_done; + /* write_zeroes/unmap journaled as all-zero page appends (no payload) */ + bool zeroes; + struct spdk_bs_io_opts io_opts; /* caller routing, per page */ + struct spdk_bs_dev_cb_args *cb_args; + TAILQ_ENTRY(md_journal_append_op) link; +}; + +struct md_journal_read_ctx { + struct spdk_bs_md_journal *jr; + /* destination description for the overlay */ + void *payload; + struct iovec *iov; + int iovcnt; + uint64_t lba; + uint32_t lba_count; + /* issue-time snapshot of the dictionary hits (spec §7): the drain + * may complete a home write and recycle the slot while the home + * read is still in flight, so the newest page copies are captured + * when the read is issued, not looked up on completion */ + uint8_t **hit_pages; /* [num_pages], NULL = home */ + uint32_t num_pages; + struct spdk_bs_dev_cb_args *orig_cb_args; + struct spdk_bs_dev_cb_args shim_cb_args; +}; + +struct spdk_bs_md_journal { + struct spdk_bs_dev *base; + struct spdk_io_channel *ch; /* base-dev channel, md thread */ + struct spdk_thread *md_thread; + + uint64_t journal_start_lba; /* base-dev blocks */ + uint32_t blocks_per_page; /* base-dev blocks per 4K page */ + uint64_t md_limit_lba; /* 0 = interception off */ + + /* ring state; head == next append slot, tail == next drain slot. + * mem_* leads (assigned at issue), disk_* mirrors on-IO-completion + * state — the two pointer pairs of the specification. */ + uint32_t mem_head, mem_tail; + uint32_t disk_head, disk_tail; + uint32_t used_slots; /* mem view */ + + /* in-memory copy of every journaled-but-undrained page (slot-indexed, + * DMA-able: it is the source of the drain home write) + header meta */ + uint8_t *page_buf; /* NUM_SLOTS * 4K */ + struct md_journal_entry_hdr *slot_hdr; /* NUM_SLOTS */ + /* lba -> newest slot dictionary, guarded by lock (spec: reads race + * with the drain thread) */ + struct md_journal_dict_bucket *dict; + uint32_t dict_tombstones; + struct spdk_spinlock lock; + + /* append pipeline: strict FIFO, one journal write in flight (I3) */ + TAILQ_HEAD(, md_journal_append_op) append_queue; + bool append_inflight; + uint8_t *hdr_dma; /* one 4K header staging block */ + /* cb_args must stay valid until IO completion; one append IO in + * flight -> one embedded instance */ + struct spdk_bs_dev_cb_args append_cb_args; + /* iov array of the in-flight append write: base devs may consume the + * iovs long after submit (the distrib bdev copies on its own poller + * thread), so it must not live on the submitting stack */ + struct iovec append_iov[2]; + + /* drain state: one entry in flight */ + struct spdk_poller *drain_poller; + bool drain_paused; /* test hook */ + /* set while this node is not the lvstore leader: the ring belongs to + * whoever is, so this process must not write to the shared device */ + bool drain_demoted; + /* waits for the append/drain pipeline to quiesce before a rescan */ + struct spdk_poller *rescan_poller; + bool drain_inflight; + struct spdk_bs_dev_cb_args drain_cb_args; + bool stopping; + bool destroy_pending; + + /* start (format/recovery) state */ + bs_md_journal_start_cb start_cb; + void *start_cb_arg; + uint8_t *recovery_buf; /* QDEPTH * 64K */ + uint32_t recovery_next_chunk; + uint32_t recovery_chunks_done; + uint32_t recovery_inflight; + int recovery_rc; + uint8_t *entry_valid; /* NUM_SLOTS bool */ + + struct spdk_bs_dev proxy; /* what the blobstore sees */ +}; + +#define __proxy_to_journal(d) SPDK_CONTAINEROF(d, struct spdk_bs_md_journal, proxy) + +static void md_journal_append_pump(struct spdk_bs_md_journal *jr); +static int md_journal_drain_poll(void *arg); +static void md_journal_finish_destroy(struct spdk_bs_md_journal *jr); + +/* called from IO completion paths: if a deferred destroy is pending and no + * base-dev IO remains in flight, finish the teardown. Returns true when the + * journal was freed (caller must not touch it). */ +static bool +md_journal_check_destroy(struct spdk_bs_md_journal *jr) +{ + if (!jr->destroy_pending) { + return false; + } + if (jr->drain_inflight || jr->append_inflight || jr->recovery_inflight != 0) { + return false; + } + md_journal_finish_destroy(jr); + return true; +} + +/* ---------------------------------------------------------------------- */ +/* dictionary (linear probe, tombstones; all under jr->lock) */ + +static uint32_t +dict_hash(uint64_t lba) +{ + /* splitmix64 finalizer */ + uint64_t z = lba + 0x9e3779b97f4a7c15ULL; + + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL; + z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL; + return (uint32_t)(z ^ (z >> 31)) & (DICT_NUM_BUCKETS - 1); +} + +static void +dict_put(struct spdk_bs_md_journal *jr, uint64_t lba, uint32_t slot) +{ + uint32_t i = dict_hash(lba); + uint32_t first_tomb = UINT32_MAX; + + for (;;) { + struct md_journal_dict_bucket *b = &jr->dict[i]; + + if (b->lba == lba) { + b->slot = slot; + return; + } + if (b->lba == DICT_TOMBSTONE_KEY && first_tomb == UINT32_MAX) { + first_tomb = i; + } else if (b->lba == DICT_EMPTY_KEY) { + if (first_tomb != UINT32_MAX) { + i = first_tomb; + jr->dict_tombstones--; + } + jr->dict[i].lba = lba; + jr->dict[i].slot = slot; + return; + } + i = (i + 1) & (DICT_NUM_BUCKETS - 1); + } +} + +static uint32_t +dict_get(struct spdk_bs_md_journal *jr, uint64_t lba) +{ + uint32_t i = dict_hash(lba); + + for (;;) { + struct md_journal_dict_bucket *b = &jr->dict[i]; + + if (b->lba == lba) { + return b->slot; + } + if (b->lba == DICT_EMPTY_KEY) { + return JOURNAL_SLOT_INVALID; + } + i = (i + 1) & (DICT_NUM_BUCKETS - 1); + } +} + +/* remove only if the mapping still points at @slot (a newer entry for the + * same lba further up the ring must keep serving reads) */ +static void +dict_remove_if_slot(struct spdk_bs_md_journal *jr, uint64_t lba, uint32_t slot) +{ + uint32_t i = dict_hash(lba); + + for (;;) { + struct md_journal_dict_bucket *b = &jr->dict[i]; + + if (b->lba == lba) { + if (b->slot == slot) { + b->lba = DICT_TOMBSTONE_KEY; + jr->dict_tombstones++; + } + return; + } + if (b->lba == DICT_EMPTY_KEY) { + return; + } + i = (i + 1) & (DICT_NUM_BUCKETS - 1); + } +} + +static void +dict_reset(struct spdk_bs_md_journal *jr) +{ + uint32_t i; + + for (i = 0; i < DICT_NUM_BUCKETS; i++) { + jr->dict[i].lba = DICT_EMPTY_KEY; + jr->dict[i].slot = JOURNAL_SLOT_INVALID; + } + jr->dict_tombstones = 0; +} + +/* ---------------------------------------------------------------------- */ +/* geometry helpers */ + +static uint64_t +slot_to_lba(struct spdk_bs_md_journal *jr, uint32_t slot) +{ + return jr->journal_start_lba + (uint64_t)slot * 2 * jr->blocks_per_page; +} + +static uint8_t * +slot_page(struct spdk_bs_md_journal *jr, uint32_t slot) +{ + return jr->page_buf + (uint64_t)slot * BS_MD_JOURNAL_PAGE_SIZE; +} + +static inline uint32_t +ring_next(uint32_t slot) +{ + return (slot + 1) % BS_MD_JOURNAL_NUM_SLOTS; +} + +static bool +ring_full(struct spdk_bs_md_journal *jr) +{ + /* one guard slot keeps a completely-valid ring unambiguous */ + return jr->used_slots >= BS_MD_JOURNAL_NUM_SLOTS - 1; +} + +/* ---------------------------------------------------------------------- */ +/* append path (spec §5): FIFO, ack caller on journal-write completion */ + +static void +append_op_complete(struct spdk_bs_md_journal *jr, struct md_journal_append_op *op, int rc) +{ + struct spdk_bs_dev_cb_args *cb_args = op->cb_args; + + free(op); + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, rc); +} + +/* copy 4K page @page_idx out of the op's payload/iov into @dst */ +static void +append_op_copy_page(struct md_journal_append_op *op, uint32_t page_idx, uint8_t *dst) +{ + uint64_t skip, copied = 0, len, n; + int i; + + if (op->zeroes) { + memset(dst, 0, BS_MD_JOURNAL_PAGE_SIZE); + return; + } + + if (op->payload != NULL) { + memcpy(dst, (uint8_t *)op->payload + (uint64_t)page_idx * BS_MD_JOURNAL_PAGE_SIZE, + BS_MD_JOURNAL_PAGE_SIZE); + return; + } + + skip = (uint64_t)page_idx * BS_MD_JOURNAL_PAGE_SIZE; + for (i = 0; i < op->iovcnt && copied < BS_MD_JOURNAL_PAGE_SIZE; i++) { + len = op->iov[i].iov_len; + if (skip >= len) { + skip -= len; + continue; + } + n = spdk_min(len - skip, BS_MD_JOURNAL_PAGE_SIZE - copied); + memcpy(dst + copied, (uint8_t *)op->iov[i].iov_base + skip, n); + copied += n; + skip = 0; + } + assert(copied == BS_MD_JOURNAL_PAGE_SIZE); +} + +static void +append_write_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct spdk_bs_md_journal *jr = cb_arg; + struct md_journal_append_op *op = TAILQ_FIRST(&jr->append_queue); + uint32_t slot; + + assert(op != NULL); + jr->append_inflight = false; + + if (jr->stopping) { + TAILQ_REMOVE(&jr->append_queue, op, link); + append_op_complete(jr, op, -ESHUTDOWN); + md_journal_check_destroy(jr); + return; + } + + if (bserrno != 0) { + SPDK_ERRLOG("md journal append failed: %d\n", bserrno); + /* the slot was never acknowledged; roll the head back */ + spdk_spin_lock(&jr->lock); + jr->mem_head = (jr->mem_head + BS_MD_JOURNAL_NUM_SLOTS - 1) % BS_MD_JOURNAL_NUM_SLOTS; + jr->used_slots--; + spdk_spin_unlock(&jr->lock); + TAILQ_REMOVE(&jr->append_queue, op, link); + append_op_complete(jr, op, bserrno); + md_journal_append_pump(jr); + return; + } + + /* the entry just written is the one before mem_head */ + slot = (jr->mem_head + BS_MD_JOURNAL_NUM_SLOTS - 1) % BS_MD_JOURNAL_NUM_SLOTS; + + /* spec §5(2): keep the page in memory and index it, only then ack */ + spdk_spin_lock(&jr->lock); + dict_put(jr, jr->slot_hdr[slot].target_lba, slot); + jr->disk_head = jr->mem_head; + spdk_spin_unlock(&jr->lock); + + op->blocks_done += jr->blocks_per_page; + if (op->blocks_done >= op->lba_count) { + TAILQ_REMOVE(&jr->append_queue, op, link); + append_op_complete(jr, op, 0); + } + md_journal_append_pump(jr); +} + +static void +md_journal_append_pump(struct spdk_bs_md_journal *jr) +{ + struct md_journal_append_op *op; + struct md_journal_entry_hdr *hdr; + uint32_t slot, page_idx; + uint8_t *page; + + if (jr->append_inflight || jr->stopping) { + return; + } + op = TAILQ_FIRST(&jr->append_queue); + if (op == NULL) { + return; + } + if (ring_full(jr)) { + /* spec: wait for the drain thread to free a slot */ + return; + } + + page_idx = op->blocks_done / jr->blocks_per_page; + slot = jr->mem_head; + + /* stage the page copy first — it doubles as the in-memory copy and + * as the (DMA-able) source of the journal write payload block */ + page = slot_page(jr, slot); + append_op_copy_page(op, page_idx, page); + + hdr = &jr->slot_hdr[slot]; + hdr->magic = BS_MD_JOURNAL_HDR_MAGIC; + hdr->crc = spdk_crc32c_update(page, BS_MD_JOURNAL_PAGE_SIZE, 0); + hdr->target_lba = op->lba + (uint64_t)page_idx * jr->blocks_per_page; + hdr->io_priority = op->io_opts.priority; + hdr->io_geometry = op->io_opts.geometry; + hdr->io_special = op->io_opts.special_io; + hdr->io_rsvd = 0; + + memset(jr->hdr_dma, 0, BS_MD_JOURNAL_PAGE_SIZE); + memcpy(jr->hdr_dma, hdr, sizeof(*hdr)); + + spdk_spin_lock(&jr->lock); + jr->mem_head = ring_next(jr->mem_head); + jr->used_slots++; + spdk_spin_unlock(&jr->lock); + jr->append_inflight = true; + + jr->append_iov[0].iov_base = jr->hdr_dma; + jr->append_iov[0].iov_len = BS_MD_JOURNAL_PAGE_SIZE; + jr->append_iov[1].iov_base = page; + jr->append_iov[1].iov_len = BS_MD_JOURNAL_PAGE_SIZE; + + jr->append_cb_args.cb_fn = append_write_cpl; + jr->append_cb_args.channel = jr->ch; + jr->append_cb_args.cb_arg = jr; + /* single 8K IO: [header][page] */ + jr->base->writev(jr->base, jr->ch, jr->append_iov, 2, slot_to_lba(jr, slot), + 2 * jr->blocks_per_page, &jr->append_cb_args, &g_ring_io_opts); +} + +static void +md_journal_append(struct spdk_bs_md_journal *jr, void *payload, struct iovec *iov, int iovcnt, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts, bool zeroes) +{ + struct md_journal_append_op *op; + + op = calloc(1, sizeof(*op)); + if (op == NULL) { + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, -ENOMEM); + return; + } + op->payload = payload; + op->iov = iov; + op->iovcnt = iovcnt; + op->zeroes = zeroes; + op->lba = lba; + op->lba_count = lba_count; + if (bs_io_opts != NULL) { + op->io_opts = *bs_io_opts; + } + op->cb_args = cb_args; + TAILQ_INSERT_TAIL(&jr->append_queue, op, link); + md_journal_append_pump(jr); +} + +/* ---------------------------------------------------------------------- */ +/* drain (spec §6): home write from the in-memory copy, then zero entry */ + +static void +drain_zero_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct spdk_bs_md_journal *jr = cb_arg; + uint32_t slot = jr->mem_tail; + + jr->drain_inflight = false; + if (jr->stopping) { + md_journal_check_destroy(jr); + return; + } + if (bserrno != 0) { + SPDK_ERRLOG("md journal entry zeroing failed: %d (retry)\n", bserrno); + return; + } + + spdk_spin_lock(&jr->lock); + dict_remove_if_slot(jr, jr->slot_hdr[slot].target_lba, slot); + jr->slot_hdr[slot].magic = 0; + jr->mem_tail = ring_next(jr->mem_tail); + jr->disk_tail = jr->mem_tail; + jr->used_slots--; + spdk_spin_unlock(&jr->lock); + + /* a slot was freed: unblock a waiting append */ + md_journal_append_pump(jr); +} + +static void +drain_home_write_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct spdk_bs_md_journal *jr = cb_arg; + + if (jr->stopping) { + jr->drain_inflight = false; + md_journal_check_destroy(jr); + return; + } + if (bserrno != 0) { + SPDK_ERRLOG("md journal home write failed: %d (retry)\n", bserrno); + jr->drain_inflight = false; + return; + } + + /* spec §6(2): unmap (zero) both blocks of the entry */ + jr->drain_cb_args.cb_fn = drain_zero_cpl; + jr->drain_cb_args.channel = jr->ch; + jr->drain_cb_args.cb_arg = jr; + jr->base->write_zeroes(jr->base, jr->ch, slot_to_lba(jr, jr->mem_tail), + 2 * jr->blocks_per_page, &jr->drain_cb_args, &g_ring_io_opts); +} + +static int +md_journal_drain_poll(void *arg) +{ + struct spdk_bs_md_journal *jr = arg; + uint32_t slot; + bool superseded; + + if (jr->drain_inflight || jr->stopping || jr->drain_paused || + jr->drain_demoted) { + return SPDK_POLLER_IDLE; + } + /* only entries whose journal write has completed (disk_head) may be + * drained — writing home before the log entry is durable would + * reintroduce the torn-write hole (I1/I2) */ + if (jr->mem_tail == jr->disk_head) { + return SPDK_POLLER_IDLE; + } + + slot = jr->mem_tail; + jr->drain_inflight = true; + + /* coalescing: if a newer copy of this lba sits further up the ring, + * skip the home write — the newer entry will cover it (still FIFO + * per lba) — and just zero this entry. */ + spdk_spin_lock(&jr->lock); + superseded = dict_get(jr, jr->slot_hdr[slot].target_lba) != slot; + spdk_spin_unlock(&jr->lock); + + if (superseded) { + jr->drain_cb_args.cb_fn = drain_zero_cpl; + jr->drain_cb_args.channel = jr->ch; + jr->drain_cb_args.cb_arg = jr; + jr->base->write_zeroes(jr->base, jr->ch, slot_to_lba(jr, slot), + 2 * jr->blocks_per_page, &jr->drain_cb_args, + &g_ring_io_opts); + return SPDK_POLLER_BUSY; + } + + { + /* replay the originating write's routing (consumed synchronously + * at submit, a stack copy is fine — same pattern as request.c) */ + struct spdk_bs_io_opts home_opts = { + .priority = jr->slot_hdr[slot].io_priority, + .geometry = jr->slot_hdr[slot].io_geometry, + .special_io = jr->slot_hdr[slot].io_special, + }; + + jr->drain_cb_args.cb_fn = drain_home_write_cpl; + jr->drain_cb_args.channel = jr->ch; + jr->drain_cb_args.cb_arg = jr; + jr->base->write(jr->base, jr->ch, slot_page(jr, slot), + jr->slot_hdr[slot].target_lba, jr->blocks_per_page, + &jr->drain_cb_args, &home_opts); + } + return SPDK_POLLER_BUSY; +} + +/* ---------------------------------------------------------------------- */ +/* read overlay (spec §7) */ + +static void +overlay_copy_page(struct md_journal_read_ctx *ctx, uint32_t page_idx, const uint8_t *src) +{ + uint64_t skip, copied = 0, len, n; + int i; + + if (ctx->payload != NULL) { + memcpy((uint8_t *)ctx->payload + (uint64_t)page_idx * BS_MD_JOURNAL_PAGE_SIZE, src, + BS_MD_JOURNAL_PAGE_SIZE); + return; + } + + skip = (uint64_t)page_idx * BS_MD_JOURNAL_PAGE_SIZE; + for (i = 0; i < ctx->iovcnt && copied < BS_MD_JOURNAL_PAGE_SIZE; i++) { + len = ctx->iov[i].iov_len; + if (skip >= len) { + skip -= len; + continue; + } + n = spdk_min(len - skip, BS_MD_JOURNAL_PAGE_SIZE - copied); + memcpy((uint8_t *)ctx->iov[i].iov_base + skip, src + copied, n); + copied += n; + skip = 0; + } +} + +static void +overlay_ctx_free(struct md_journal_read_ctx *ctx) +{ + uint32_t i; + + if (ctx->hit_pages != NULL) { + for (i = 0; i < ctx->num_pages; i++) { + free(ctx->hit_pages[i]); + } + free(ctx->hit_pages); + } + free(ctx); +} + +static void +overlay_read_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct md_journal_read_ctx *ctx = cb_arg; + struct spdk_bs_dev_cb_args *orig = ctx->orig_cb_args; + uint32_t i; + + if (bserrno == 0) { + for (i = 0; i < ctx->num_pages; i++) { + if (ctx->hit_pages[i] != NULL) { + overlay_copy_page(ctx, i, ctx->hit_pages[i]); + } + } + } + + overlay_ctx_free(ctx); + orig->cb_fn(orig->channel, orig->cb_arg, bserrno); +} + +static struct md_journal_read_ctx * +overlay_ctx_create(struct spdk_bs_md_journal *jr, void *payload, struct iovec *iov, int iovcnt, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args) +{ + struct md_journal_read_ctx *ctx; + uint32_t i; + + ctx = calloc(1, sizeof(*ctx)); + if (ctx == NULL) { + return NULL; + } + ctx->jr = jr; + ctx->payload = payload; + ctx->iov = iov; + ctx->iovcnt = iovcnt; + ctx->lba = lba; + ctx->lba_count = lba_count; + ctx->num_pages = lba_count / jr->blocks_per_page; + ctx->orig_cb_args = cb_args; + ctx->shim_cb_args.cb_fn = overlay_read_cpl; + ctx->shim_cb_args.channel = cb_args->channel; + ctx->shim_cb_args.cb_arg = ctx; + + ctx->hit_pages = calloc(ctx->num_pages, sizeof(*ctx->hit_pages)); + if (ctx->hit_pages == NULL) { + free(ctx); + return NULL; + } + + /* snapshot every dictionary hit now — the slot may be drained and + * recycled before the home read completes, and the home read may + * return the pre-drain content of the page */ + spdk_spin_lock(&jr->lock); + for (i = 0; i < ctx->num_pages; i++) { + uint32_t slot = dict_get(jr, lba + (uint64_t)i * jr->blocks_per_page); + + if (slot == JOURNAL_SLOT_INVALID) { + continue; + } + ctx->hit_pages[i] = malloc(BS_MD_JOURNAL_PAGE_SIZE); + if (ctx->hit_pages[i] == NULL) { + spdk_spin_unlock(&jr->lock); + overlay_ctx_free(ctx); + return NULL; + } + memcpy(ctx->hit_pages[i], slot_page(jr, slot), BS_MD_JOURNAL_PAGE_SIZE); + } + spdk_spin_unlock(&jr->lock); + return ctx; +} + +/* ---------------------------------------------------------------------- */ +/* proxy bs_dev */ + +static inline bool +lba_is_md(struct spdk_bs_md_journal *jr, uint64_t lba, uint64_t lba_count) +{ + return jr->md_limit_lba != 0 && lba + lba_count <= jr->md_limit_lba; +} + +static struct spdk_io_channel * +proxy_create_channel(struct spdk_bs_dev *dev) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + return jr->base->create_channel(jr->base); +} + +static void +proxy_destroy_channel(struct spdk_bs_dev *dev, struct spdk_io_channel *channel) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + jr->base->destroy_channel(jr->base, channel); +} + +static void +proxy_destroy(struct spdk_bs_dev *dev) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + jr->stopping = true; + spdk_poller_unregister(&jr->drain_poller); + if (jr->drain_inflight || jr->append_inflight || jr->recovery_inflight != 0) { + /* an IO against the base dev is still in flight; its completion + * callback finishes the teardown (md_journal_check_destroy) */ + jr->destroy_pending = true; + return; + } + md_journal_finish_destroy(jr); +} + +static void +proxy_read(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, void *payload, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + struct md_journal_read_ctx *ctx = + overlay_ctx_create(jr, payload, NULL, 0, lba, lba_count, cb_args); + + if (ctx == NULL) { + /* a raw fallback read could serve pages the journal + * has not written home yet — fail instead */ + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, -ENOMEM); + return; + } + jr->base->read(jr->base, channel, payload, lba, lba_count, &ctx->shim_cb_args, + bs_io_opts); + return; + } + jr->base->read(jr->base, channel, payload, lba, lba_count, cb_args, bs_io_opts); +} + +static void +proxy_write(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, void *payload, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + md_journal_append(jr, payload, NULL, 0, lba, lba_count, cb_args, bs_io_opts, false); + return; + } + jr->base->write(jr->base, channel, payload, lba, lba_count, cb_args, bs_io_opts); +} + +static void +proxy_readv(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + struct md_journal_read_ctx *ctx = + overlay_ctx_create(jr, NULL, iov, iovcnt, lba, lba_count, cb_args); + + if (ctx == NULL) { + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, -ENOMEM); + return; + } + jr->base->readv(jr->base, channel, iov, iovcnt, lba, lba_count, + &ctx->shim_cb_args, bs_io_opts); + return; + } + jr->base->readv(jr->base, channel, iov, iovcnt, lba, lba_count, cb_args, bs_io_opts); +} + +static void +proxy_writev(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + md_journal_append(jr, NULL, iov, iovcnt, lba, lba_count, cb_args, bs_io_opts, false); + return; + } + jr->base->writev(jr->base, channel, iov, iovcnt, lba, lba_count, cb_args, bs_io_opts); +} + +static void +proxy_readv_ext(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_blob_ext_io_opts *ext_opts, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + struct md_journal_read_ctx *ctx = + overlay_ctx_create(jr, NULL, iov, iovcnt, lba, lba_count, cb_args); + + if (ctx == NULL) { + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, -ENOMEM); + return; + } + jr->base->readv_ext(jr->base, channel, iov, iovcnt, lba, lba_count, + &ctx->shim_cb_args, ext_opts, bs_io_opts); + return; + } + jr->base->readv_ext(jr->base, channel, iov, iovcnt, lba, lba_count, cb_args, ext_opts, + bs_io_opts); +} + +static void +proxy_writev_ext(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_blob_ext_io_opts *ext_opts, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (lba_is_md(jr, lba, lba_count)) { + md_journal_append(jr, NULL, iov, iovcnt, lba, lba_count, cb_args, bs_io_opts, false); + return; + } + jr->base->writev_ext(jr->base, channel, iov, iovcnt, lba, lba_count, cb_args, ext_opts, + bs_io_opts); +} + +static void +proxy_flush(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct spdk_bs_dev_cb_args *cb_args) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + jr->base->flush(jr->base, channel, cb_args); +} + +static void +proxy_write_zeroes(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + uint64_t lba, uint64_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + /* md-range zeroing (e.g. blob delete / md-chain release zeroing pages + * in place, blob_persist_zero_pages) MUST take the same FIFO path as + * journaled page writes: a raw passthrough races the deferred home + * write of an older journaled copy of the same page, and the drain + * would then resurrect the zeroed page on disk. Journal it as an + * append of all-zero pages. */ + if (lba_is_md(jr, lba, lba_count) && + lba % jr->blocks_per_page == 0 && lba_count % jr->blocks_per_page == 0) { + md_journal_append(jr, NULL, NULL, 0, lba, (uint32_t)lba_count, cb_args, + bs_io_opts, true); + return; + } + jr->base->write_zeroes(jr->base, channel, lba, lba_count, cb_args, bs_io_opts); +} + +static void +proxy_unmap(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + uint64_t lba, uint64_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + /* same ordering hazard as proxy_write_zeroes: unmapped md ranges read + * back as zeros, so journaling an all-zero page append is equivalent + * and keeps FIFO order with pending journaled writes of the page */ + if (lba_is_md(jr, lba, lba_count) && + lba % jr->blocks_per_page == 0 && lba_count % jr->blocks_per_page == 0) { + md_journal_append(jr, NULL, NULL, 0, lba, (uint32_t)lba_count, cb_args, + bs_io_opts, true); + return; + } + jr->base->unmap(jr->base, channel, lba, lba_count, cb_args, bs_io_opts); +} + +static struct spdk_bdev * +proxy_get_base_bdev(struct spdk_bs_dev *dev) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + if (jr->base->get_base_bdev != NULL) { + return jr->base->get_base_bdev(jr->base); + } + return NULL; +} + +static bool +proxy_is_zeroes(struct spdk_bs_dev *dev, uint64_t lba, uint64_t lba_count) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + return jr->base->is_zeroes(jr->base, lba, lba_count); +} + +static bool +proxy_is_range_valid(struct spdk_bs_dev *dev, uint64_t lba, uint64_t lba_count) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + return jr->base->is_range_valid(jr->base, lba, lba_count); +} + +static bool +proxy_translate_lba(struct spdk_bs_dev *dev, uint64_t lba, uint64_t *base_lba) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + return jr->base->translate_lba(jr->base, lba, base_lba); +} + +static void +proxy_copy(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + uint64_t dst_lba, uint64_t src_lba, uint64_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_bs_io_opts *bs_io_opts) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + jr->base->copy(jr->base, channel, dst_lba, src_lba, lba_count, cb_args, bs_io_opts); +} + +static bool +proxy_is_degraded(struct spdk_bs_dev *dev) +{ + struct spdk_bs_md_journal *jr = __proxy_to_journal(dev); + + return jr->base->is_degraded(jr->base); +} + +/* ---------------------------------------------------------------------- */ +/* start: fresh format (zero ring) or recovery scan (spec §8) */ + +static void +start_finish(struct spdk_bs_md_journal *jr, int rc) +{ + bs_md_journal_start_cb cb = jr->start_cb; + void *cb_arg = jr->start_cb_arg; + + spdk_free(jr->recovery_buf); + jr->recovery_buf = NULL; + free(jr->entry_valid); + jr->entry_valid = NULL; + jr->start_cb = NULL; + + if (rc == 0) { + jr->drain_poller = SPDK_POLLER_REGISTER(md_journal_drain_poll, jr, 0); + } + if (cb != NULL) { + cb(cb_arg, rc); + } +} + +uint64_t +bs_md_journal_ring_lba(struct spdk_bs_md_journal *jr) +{ + return jr->journal_start_lba; +} + +uint64_t +bs_md_journal_ring_lba_count(struct spdk_bs_md_journal *jr) +{ + return (BS_MD_JOURNAL_SIZE_BYTES / BS_MD_JOURNAL_PAGE_SIZE) * jr->blocks_per_page; +} + +static void +recovery_rebuild(struct spdk_bs_md_journal *jr) +{ + uint32_t n = BS_MD_JOURNAL_NUM_SLOTS; + uint32_t first_empty = UINT32_MAX; + uint32_t i, s, run_len = 0, valid_total = 0; + + for (i = 0; i < n; i++) { + if (!jr->entry_valid[i] && first_empty == UINT32_MAX) { + first_empty = i; + } + valid_total += jr->entry_valid[i]; + } + + if (valid_total == 0) { + SPDK_NOTICELOG("md journal recovery: ring empty\n"); + start_finish(jr, 0); + return; + } + if (first_empty == UINT32_MAX) { + /* cannot happen with the guard slot */ + SPDK_ERRLOG("md journal recovery: ring fully valid, order ambiguous\n"); + start_finish(jr, -EIO); + return; + } + + /* the valid entries form one contiguous run in ring order (zero-on- + * drain guarantees it): walk from the first empty slot forward, the + * run start is the recovered tail. */ + s = first_empty; + while (!jr->entry_valid[s]) { + s = ring_next(s); + if (s == first_empty) { + break; + } + } + + spdk_spin_lock(&jr->lock); + jr->mem_tail = jr->disk_tail = s; + for (i = s; jr->entry_valid[i]; i = ring_next(i)) { + dict_put(jr, jr->slot_hdr[i].target_lba, i); /* FIFO: later wins */ + run_len++; + } + jr->mem_head = jr->disk_head = i; + jr->used_slots = run_len; + spdk_spin_unlock(&jr->lock); + + if (run_len != valid_total) { + SPDK_ERRLOG("md journal recovery: %u valid entries outside the contiguous " + "run of %u are treated as empty\n", valid_total - run_len, run_len); + } + SPDK_NOTICELOG("md journal recovery: %u entries to drain (tail=%u head=%u)\n", + run_len, jr->mem_tail, jr->mem_head); + start_finish(jr, 0); +} + +static void recovery_issue_next(struct spdk_bs_md_journal *jr, uint32_t buf_idx); + +static void +recovery_chunk_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct md_journal_read_ctx *rctx = cb_arg; + struct spdk_bs_md_journal *jr = rctx->jr; + uint32_t buf_idx = (uint32_t)rctx->lba_count; /* stashed buffer index */ + uint32_t chunk = (uint32_t)rctx->lba; /* stashed chunk index */ + uint32_t entries_per_chunk = BS_MD_JOURNAL_RECOVERY_IO_SIZE / BS_MD_JOURNAL_ENTRY_BYTES; + uint8_t *buf = jr->recovery_buf + (uint64_t)buf_idx * BS_MD_JOURNAL_RECOVERY_IO_SIZE; + uint32_t i; + + free(rctx); + jr->recovery_inflight--; + jr->recovery_chunks_done++; + + if (jr->stopping) { + if (!md_journal_check_destroy(jr) && jr->recovery_inflight == 0) { + start_finish(jr, -ESHUTDOWN); + } + return; + } + + if (bserrno != 0) { + SPDK_ERRLOG("md journal recovery read (chunk %u) failed: %d\n", chunk, bserrno); + jr->recovery_rc = bserrno; + } else if (jr->recovery_rc == 0) { + for (i = 0; i < entries_per_chunk; i++) { + uint32_t slot = chunk * entries_per_chunk + i; + struct md_journal_entry_hdr hdr; + uint8_t *entry = buf + (uint64_t)i * BS_MD_JOURNAL_ENTRY_BYTES; + uint8_t *page = entry + BS_MD_JOURNAL_PAGE_SIZE; + + memcpy(&hdr, entry, sizeof(hdr)); + /* zeroed and corrupted entries are both empty */ + if (hdr.magic != BS_MD_JOURNAL_HDR_MAGIC || + spdk_crc32c_update(page, BS_MD_JOURNAL_PAGE_SIZE, 0) != hdr.crc) { + continue; + } + jr->entry_valid[slot] = 1; + jr->slot_hdr[slot] = hdr; + memcpy(slot_page(jr, slot), page, BS_MD_JOURNAL_PAGE_SIZE); + } + } + + recovery_issue_next(jr, buf_idx); +} + +static void +recovery_issue_next(struct spdk_bs_md_journal *jr, uint32_t buf_idx) +{ + uint32_t total_chunks = BS_MD_JOURNAL_SIZE_BYTES / BS_MD_JOURNAL_RECOVERY_IO_SIZE; + struct md_journal_read_ctx *rctx; + uint32_t chunk; + + if (jr->recovery_rc != 0 || jr->recovery_next_chunk >= total_chunks) { + if (jr->recovery_inflight == 0) { + if (jr->recovery_rc != 0) { + start_finish(jr, jr->recovery_rc); + } else { + recovery_rebuild(jr); + } + } + return; + } + + chunk = jr->recovery_next_chunk++; + rctx = calloc(1, sizeof(*rctx)); + if (rctx == NULL) { + jr->recovery_rc = -ENOMEM; + if (jr->recovery_inflight == 0) { + start_finish(jr, -ENOMEM); + } + return; + } + rctx->jr = jr; + rctx->lba = chunk; /* stash chunk index */ + rctx->lba_count = buf_idx; /* stash buffer index */ + rctx->shim_cb_args.cb_fn = recovery_chunk_cpl; + rctx->shim_cb_args.channel = jr->ch; + rctx->shim_cb_args.cb_arg = rctx; + + jr->recovery_inflight++; + jr->base->read(jr->base, jr->ch, + jr->recovery_buf + (uint64_t)buf_idx * BS_MD_JOURNAL_RECOVERY_IO_SIZE, + jr->journal_start_lba + + (uint64_t)chunk * (BS_MD_JOURNAL_RECOVERY_IO_SIZE / BS_MD_JOURNAL_PAGE_SIZE) * + jr->blocks_per_page, + (BS_MD_JOURNAL_RECOVERY_IO_SIZE / BS_MD_JOURNAL_PAGE_SIZE) * jr->blocks_per_page, + &rctx->shim_cb_args, &g_ring_io_opts); +} + +void +bs_md_journal_start(struct spdk_bs_md_journal *jr, bool fresh_format, + bs_md_journal_start_cb cb_fn, void *cb_arg) +{ + uint32_t i; + + jr->md_thread = spdk_get_thread(); + jr->start_cb = cb_fn; + jr->start_cb_arg = cb_arg; + + jr->ch = jr->base->create_channel(jr->base); + if (jr->ch == NULL) { + jr->start_cb = NULL; + cb_fn(cb_arg, -ENOMEM); + return; + } + + if (fresh_format) { + /* the ring itself is zeroed by the caller's init batch + * (bs_md_journal_ring_lba/_count); nothing to read back */ + start_finish(jr, 0); + return; + } + + /* Arm interception for the whole proxy range before the caller + * issues its first md read: the super block is read before the + * metadata layout is known, and after a crash its newest version + * may still sit in the ring (torn or stale home copy). Overlaying + * is correct for any LBA — a dictionary miss passes through — and + * the caller tightens the limit once the super is parsed. */ + bs_md_journal_enable(jr, jr->journal_start_lba); + + /* recovery: read the whole ring with up to 32 parallel 64K IOs */ + jr->entry_valid = calloc(BS_MD_JOURNAL_NUM_SLOTS, 1); + jr->recovery_buf = spdk_zmalloc((uint64_t)BS_MD_JOURNAL_RECOVERY_QDEPTH * + BS_MD_JOURNAL_RECOVERY_IO_SIZE, + BS_MD_JOURNAL_PAGE_SIZE, NULL, + SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA); + if (jr->entry_valid == NULL || jr->recovery_buf == NULL) { + start_finish(jr, -ENOMEM); + return; + } + jr->recovery_next_chunk = 0; + jr->recovery_chunks_done = 0; + jr->recovery_inflight = 0; + jr->recovery_rc = 0; + for (i = 0; i < BS_MD_JOURNAL_RECOVERY_QDEPTH; i++) { + recovery_issue_next(jr, i); + } +} + +/* ---------------------------------------------------------------------- */ +/* rescan: recovery on an already-started journal (peer takeover) */ + +static int +md_journal_rescan_start(struct spdk_bs_md_journal *jr) +{ + uint32_t i; + + jr->entry_valid = calloc(BS_MD_JOURNAL_NUM_SLOTS, 1); + jr->recovery_buf = spdk_zmalloc((uint64_t)BS_MD_JOURNAL_RECOVERY_QDEPTH * + BS_MD_JOURNAL_RECOVERY_IO_SIZE, + BS_MD_JOURNAL_PAGE_SIZE, NULL, + SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA); + if (jr->entry_valid == NULL || jr->recovery_buf == NULL) { + start_finish(jr, -ENOMEM); + return -ENOMEM; + } + + /* drop the view built at load time: every pointer and every cached + * page is re-derived from the ring below */ + spdk_spin_lock(&jr->lock); + dict_reset(jr); + jr->mem_head = jr->mem_tail = jr->disk_head = jr->disk_tail = 0; + jr->used_slots = 0; + spdk_spin_unlock(&jr->lock); + + jr->recovery_next_chunk = 0; + jr->recovery_chunks_done = 0; + jr->recovery_inflight = 0; + jr->recovery_rc = 0; + SPDK_NOTICELOG("md journal rescan: re-reading the ring on takeover\n"); + for (i = 0; i < BS_MD_JOURNAL_RECOVERY_QDEPTH; i++) { + recovery_issue_next(jr, i); + } + return 0; +} + +static int +md_journal_rescan_quiesce_poll(void *arg) +{ + struct spdk_bs_md_journal *jr = arg; + + if (jr->stopping || jr->destroy_pending) { + spdk_poller_unregister(&jr->rescan_poller); + start_finish(jr, -ESHUTDOWN); + return SPDK_POLLER_BUSY; + } + if (jr->drain_inflight || jr->append_inflight || + !TAILQ_EMPTY(&jr->append_queue)) { + return SPDK_POLLER_IDLE; + } + spdk_poller_unregister(&jr->rescan_poller); + md_journal_rescan_start(jr); + return SPDK_POLLER_BUSY; +} + +void +bs_md_journal_rescan(struct spdk_bs_md_journal *jr, bs_md_journal_start_cb cb_fn, + void *cb_arg) +{ + assert(spdk_get_thread() == jr->md_thread); + + if (jr->stopping || jr->destroy_pending) { + cb_fn(cb_arg, -ESHUTDOWN); + return; + } + if (jr->start_cb != NULL || jr->rescan_poller != NULL) { + /* a start/rescan is already running */ + cb_fn(cb_arg, -EBUSY); + return; + } + + jr->start_cb = cb_fn; + jr->start_cb_arg = cb_arg; + + /* the drain poller must not touch the ring while it is re-read, and + * start_finish() re-registers it when the rescan completes */ + spdk_poller_unregister(&jr->drain_poller); + + if (jr->drain_inflight || jr->append_inflight || + !TAILQ_EMPTY(&jr->append_queue)) { + /* an entry write or home write is in flight: its completion + * still writes into the old buffer/pointers, so wait it out */ + jr->rescan_poller = SPDK_POLLER_REGISTER(md_journal_rescan_quiesce_poll, + jr, 200); + return; + } + md_journal_rescan_start(jr); +} + +void +bs_md_journal_set_leader(struct spdk_bs_md_journal *jr, bool leader) +{ + if (jr->drain_demoted == !leader) { + return; + } + jr->drain_demoted = !leader; + SPDK_NOTICELOG("md journal drain %s: this node is %s the lvstore leader " + "(%u entries held)\n", leader ? "resumed" : "stopped", + leader ? "again" : "no longer", jr->used_slots); +} + +void +bs_md_journal_get_stats(struct spdk_bs_md_journal *jr, bool *enabled, + uint32_t *num_slots, uint32_t *used_slots, + uint32_t *mem_head, uint32_t *mem_tail, + uint32_t *disk_head, uint32_t *disk_tail, + bool *drain_paused, bool *drain_demoted) +{ + spdk_spin_lock(&jr->lock); + *enabled = (jr->md_limit_lba != 0); + *num_slots = BS_MD_JOURNAL_NUM_SLOTS; + *used_slots = jr->used_slots; + *mem_head = jr->mem_head; + *mem_tail = jr->mem_tail; + *disk_head = jr->disk_head; + *disk_tail = jr->disk_tail; + *drain_paused = jr->drain_paused; + *drain_demoted = jr->drain_demoted; + spdk_spin_unlock(&jr->lock); +} + +void +bs_md_journal_set_drain_paused(struct spdk_bs_md_journal *jr, bool paused) +{ + SPDK_NOTICELOG("md journal drain %s\n", paused ? "PAUSED (test hook)" : "resumed"); + jr->drain_paused = paused; +} + +void +bs_md_journal_enable(struct spdk_bs_md_journal *jr, uint64_t md_limit_lba) +{ + SPDK_NOTICELOG("md journal enabled: md limit lba %" PRIu64 ", ring @ lba %" PRIu64 + " (%u slots)\n", md_limit_lba, jr->journal_start_lba, + (uint32_t)BS_MD_JOURNAL_NUM_SLOTS); + jr->md_limit_lba = md_limit_lba; +} + +static void +md_journal_finish_destroy(struct spdk_bs_md_journal *jr) +{ + struct spdk_bs_dev *base = jr->base; + + spdk_poller_unregister(&jr->drain_poller); + spdk_poller_unregister(&jr->rescan_poller); + if (jr->ch != NULL) { + base->destroy_channel(base, jr->ch); + jr->ch = NULL; + } + spdk_spin_destroy(&jr->lock); + spdk_free(jr->page_buf); + spdk_free(jr->hdr_dma); + spdk_free(jr->recovery_buf); + free(jr->entry_valid); + free(jr->slot_hdr); + free(jr->dict); + free(jr); + base->destroy(base); +} + +/* ---------------------------------------------------------------------- */ + +struct spdk_bs_dev * +bs_md_journal_dev_create(struct spdk_bs_dev *base, struct spdk_bs_md_journal **_journal) +{ + struct spdk_bs_md_journal *jr; + uint64_t journal_blocks; + + if (BS_MD_JOURNAL_PAGE_SIZE % base->blocklen != 0) { + SPDK_ERRLOG("unsupported base blocklen %u\n", base->blocklen); + return NULL; + } + journal_blocks = BS_MD_JOURNAL_SIZE_BYTES / base->blocklen; + if (base->blockcnt <= 2 * journal_blocks) { + SPDK_ERRLOG("device too small for md journal\n"); + return NULL; + } + + jr = calloc(1, sizeof(*jr)); + if (jr == NULL) { + return NULL; + } + jr->base = base; + jr->blocks_per_page = BS_MD_JOURNAL_PAGE_SIZE / base->blocklen; + /* highest offset of the actual (runtime) virtual device size */ + jr->journal_start_lba = base->blockcnt - journal_blocks; + + jr->page_buf = spdk_zmalloc((uint64_t)BS_MD_JOURNAL_NUM_SLOTS * BS_MD_JOURNAL_PAGE_SIZE, + BS_MD_JOURNAL_PAGE_SIZE, NULL, SPDK_ENV_SOCKET_ID_ANY, + SPDK_MALLOC_DMA); + jr->hdr_dma = spdk_zmalloc(BS_MD_JOURNAL_PAGE_SIZE, BS_MD_JOURNAL_PAGE_SIZE, NULL, + SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA); + jr->slot_hdr = calloc(BS_MD_JOURNAL_NUM_SLOTS, sizeof(*jr->slot_hdr)); + jr->dict = calloc(DICT_NUM_BUCKETS, sizeof(*jr->dict)); + if (jr->page_buf == NULL || jr->hdr_dma == NULL || jr->slot_hdr == NULL || + jr->dict == NULL) { + spdk_free(jr->page_buf); + spdk_free(jr->hdr_dma); + free(jr->slot_hdr); + free(jr->dict); + free(jr); + return NULL; + } + + spdk_spin_init(&jr->lock); + TAILQ_INIT(&jr->append_queue); + dict_reset(jr); + + /* the blobstore sees a device shrunk by the ring — data clusters can + * never collide with the journal region. Every callback that receives + * the dev pointer must be an explicit shim (container_of safety); the + * base vtable is never exposed directly. */ + jr->proxy.blockcnt = jr->journal_start_lba; + jr->proxy.blocklen = base->blocklen; + jr->proxy.create_channel = proxy_create_channel; + jr->proxy.destroy_channel = proxy_destroy_channel; + jr->proxy.destroy = proxy_destroy; + jr->proxy.read = proxy_read; + jr->proxy.write = proxy_write; + jr->proxy.readv = proxy_readv; + jr->proxy.writev = proxy_writev; + jr->proxy.readv_ext = base->readv_ext != NULL ? proxy_readv_ext : NULL; + jr->proxy.writev_ext = base->writev_ext != NULL ? proxy_writev_ext : NULL; + jr->proxy.flush = proxy_flush; + jr->proxy.write_zeroes = proxy_write_zeroes; + jr->proxy.unmap = proxy_unmap; + jr->proxy.get_base_bdev = base->get_base_bdev != NULL ? proxy_get_base_bdev : NULL; + jr->proxy.is_zeroes = base->is_zeroes != NULL ? proxy_is_zeroes : NULL; + jr->proxy.is_range_valid = base->is_range_valid != NULL ? proxy_is_range_valid : NULL; + jr->proxy.translate_lba = base->translate_lba != NULL ? proxy_translate_lba : NULL; + jr->proxy.copy = base->copy != NULL ? proxy_copy : NULL; + jr->proxy.is_degraded = base->is_degraded != NULL ? proxy_is_degraded : NULL; + + *_journal = jr; + return &jr->proxy; +} diff --git a/lib/blob/blob_md_journal.h b/lib/blob/blob_md_journal.h new file mode 100644 index 00000000000..59aff848ff9 --- /dev/null +++ b/lib/blob/blob_md_journal.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (C) 2026 Simplyblock GmbH. + * + * LVS metadata (page) journaling with torn-write protection. + * + * Per-LVS 64 MB journal ring reserved at the highest offset of the + * backing virtual device. Every metadata-page write is redirected to + * the ring ([header block][4K md page] per entry), acknowledged to the + * caller once the journal write is durable, mirrored into an in-memory + * buffer + LBA dictionary (read-amplification avoidance), and drained + * to its home LBA by an asynchronous poller which then zeroes the + * entry. Recovery re-reads the whole ring (<= 32 parallel 64 KB IOs), + * treats zeroed/corrupt entries as empty, rebuilds buffer/dictionary/ + * pointers and lets the drain poller work off the backlog. + * + * See blobstore_metadata_journal_design.md (rev 3). + */ + +#ifndef SPDK_BLOB_MD_JOURNAL_H +#define SPDK_BLOB_MD_JOURNAL_H + +#include "spdk/stdinc.h" +#include "spdk/blob.h" + +#define BS_MD_JOURNAL_SIZE_BYTES (64ULL * 1024 * 1024) +#define BS_MD_JOURNAL_PAGE_SIZE 4096ULL +/* One entry = header block + md-page block. */ +#define BS_MD_JOURNAL_ENTRY_BYTES (2 * BS_MD_JOURNAL_PAGE_SIZE) +#define BS_MD_JOURNAL_NUM_SLOTS (BS_MD_JOURNAL_SIZE_BYTES / BS_MD_JOURNAL_ENTRY_BYTES) +#define BS_MD_JOURNAL_RECOVERY_IO_SIZE (64ULL * 1024) +#define BS_MD_JOURNAL_RECOVERY_QDEPTH 32 +#define BS_MD_JOURNAL_HDR_MAGIC 0x4D444A31 /* "MDJ1" */ + +struct spdk_bs_md_journal; + +/* Wrap @base. The returned bs_dev reports blockcnt shrunk by 64 MB so the + * blobstore can never allocate over the ring; md-range writes are + * journaled and md-range reads are overlaid from the in-memory copies + * once the journal is enabled. *_journal receives the journal handle. + * Returns NULL on allocation failure (base is left untouched). */ +struct spdk_bs_dev *bs_md_journal_dev_create(struct spdk_bs_dev *base, + struct spdk_bs_md_journal **_journal); + +typedef void (*bs_md_journal_start_cb)(void *cb_arg, int bserrno); + +/* Asynchronous bring-up on the blobstore md thread. + * fresh_format=true (spdk_bs_init): no disk IO — the caller must zero the + * ring region (bs_md_journal_ring_lba/_count) in its init batch; completes + * inline. Interception stays off until bs_md_journal_enable. + * fresh_format=false (spdk_bs_load): run recovery (parallel scan, checksum + * validation, buffer/dictionary/pointer rebuild) and arm interception for + * the WHOLE proxy range — the super block is read before the md layout is + * known and its newest version may still sit in the ring; the caller + * tightens the range via bs_md_journal_enable once the super is parsed. + * Also creates the journal's base-dev channel and registers the drain + * poller. Must complete before any md read is issued on load. */ +void bs_md_journal_start(struct spdk_bs_md_journal *journal, bool fresh_format, + bs_md_journal_start_cb cb_fn, void *cb_arg); + +/* Ring region in base-dev blocks (above the proxy's blockcnt) — for the + * spdk_bs_init batch to zero on fresh format. */ +uint64_t bs_md_journal_ring_lba(struct spdk_bs_md_journal *journal); +uint64_t bs_md_journal_ring_lba_count(struct spdk_bs_md_journal *journal); + +/* Re-run recovery on a journal that is already started, i.e. re-read the + * ring, rebuild buffer/dictionary/pointers and let the drain poller work off + * whatever backlog another node left behind. + * + * Needed because the ring is shared state on a shared device while the + * in-memory buffer/dictionary is per process. A secondary that loaded the + * lvstore while the primary was alive scanned the ring at ITS load time; the + * entries the primary appended (and acknowledged) afterwards are invisible to + * it. Recovering only on load is therefore not enough for the product's + * failover path, which promotes an already-loaded peer + * (bdev_lvol_update_lvstore + bdev_lvol_set_leader_all) instead of loading + * the lvstore anew - without a rescan the new leader serves stale home pages + * and appends over the dead leader's undrained entries. + * + * Runs on the md thread; the caller must not have md reads outstanding. The + * rescan waits for the append/drain pipeline to quiesce first. */ +void bs_md_journal_rescan(struct spdk_bs_md_journal *journal, + bs_md_journal_start_cb cb_fn, void *cb_arg); + +/* Arm interception: every write/read whose LBA range lies entirely below + * @md_limit_lba (exclusive, in base-dev blocks) goes through the journal. + * Called once the metadata layout is known (super parsed / init layout). */ +void bs_md_journal_enable(struct spdk_bs_md_journal *journal, uint64_t md_limit_lba); + +/* Follow the lvstore's leadership: only the leader may drain. + * + * The drain poller is a background writer that the pre-journal md path did not + * have - it keeps pushing this process's in-memory pages to their home LBAs on + * the SHARED device with no IO to trigger it. A node that stops being the + * leader but stays alive therefore keeps writing metadata behind the new + * leader's back. That happens in the product on the network-outage path + * (spdk_lvs_change_leader_state / groupid 0: freeze, block_port, leader=false, + * process still running) and in the window before a writer-conflict abort + * completes. Stopping the drain on demotion is the journal's half of that + * contract; the buffer it was holding is rebuilt by bs_md_journal_rescan when + * this node is promoted again. + * + * Called from spdk_bs_set_leader(). + * + * OPEN QUESTION (phase-3 test F3b): a demoted node can still APPEND. The lvol + * layer gates destroy and async delete on lvs->leader but not create, and it + * deliberately allows a SYNC delete on a non-leader + * (rpc_bdev_lvol_delete: "Deleting async lvol on non-leader lvs" is refused, + * sync is not) - so "a non-leader never writes md" is not the fork's model. + * Entries a non-leader appends now stay in its ring (this stops it writing + * them home over the leader's data) until it is promoted and rescans, or the + * leader's own head marches over the slots. Either the lvol layer must refuse + * md-mutating work on a non-leader, or the ring needs an owner/epoch so a + * second appender is rejected by the device. Recorded in + * blobstore_metadata_journal_design.md section 11.2. */ +void bs_md_journal_set_leader(struct spdk_bs_md_journal *journal, bool leader); + +/* Ring state as this process sees it, and the drain test hook (see + * spdk_bs_md_journal_stats in include/spdk/blob.h). Pausing the drain lets a + * test accumulate acknowledged-but-not-home entries: at any md rate the + * blobstore can produce, the drain otherwise keeps the ring at ~1 entry, so + * neither recovery-with-entries nor journal-full is reachable by workload + * alone. */ +void bs_md_journal_get_stats(struct spdk_bs_md_journal *journal, bool *enabled, + uint32_t *num_slots, uint32_t *used_slots, + uint32_t *mem_head, uint32_t *mem_tail, + uint32_t *disk_head, uint32_t *disk_tail, + bool *drain_paused, bool *drain_demoted); +void bs_md_journal_set_drain_paused(struct spdk_bs_md_journal *journal, bool paused); + +/* Teardown happens through the proxy's bs_dev->destroy(): it quiesces + * in-flight journal IO, frees the journal and destroys the base dev. */ + +#endif /* SPDK_BLOB_MD_JOURNAL_H */ diff --git a/lib/blob/blobstore.c b/lib/blob/blobstore.c index c7458246bc8..4c7a44c1703 100644 --- a/lib/blob/blobstore.c +++ b/lib/blob/blobstore.c @@ -23,6 +23,7 @@ #include "spdk/log.h" #include "blobstore.h" +#include "blob_md_journal.h" #define BLOB_CRC32C_INITIAL 0xffffffffUL @@ -4676,6 +4677,10 @@ struct spdk_bs_load_ctx { bool force_recover; + /* deferred spdk_bs_load completion while the md journal recovers */ + spdk_bs_op_with_handle_complete load_cb_fn; + void *load_cb_arg; + /* These fields are used in the spdk_bs_dump path. */ bool dumping; bool snapshot_create_mode; @@ -6494,6 +6499,13 @@ bs_parse_super(struct spdk_bs_load_ctx *ctx) return -ENOMEM; } + if (ctx->bs->md_journal != NULL) { + /* metadata layout is known now — arm write/read interception + * for the whole md region (super, masks, md pages) */ + bs_md_journal_enable(ctx->bs->md_journal, + bs_page_to_lba(ctx->bs, ctx->bs->md_start + ctx->bs->md_len)); + } + ctx->bs->total_data_clusters = ctx->bs->total_clusters - spdk_divide_round_up( ctx->bs->md_start + ctx->bs->md_len, ctx->bs->pages_per_cluster); ctx->bs->super_blob = ctx->super->super_blob; @@ -6596,15 +6608,95 @@ bs_opts_print(struct spdk_bs_opts *opts) return 0; } +static void bs_load_read_super(void *cb_arg, int bserrno); + +/* Raw pre-probe of the super block: the md_journal feature flag decides + * whether the device must be wrapped with the torn-write-protection + * journal before any other metadata access (legacy stores load through + * the unwrapped path unchanged). The flag never changes over a store's + * lifetime, so even a stale or torn-on-drain raw super carries it. */ +struct bs_load_probe_ctx { + struct spdk_bs_dev *dev; + struct spdk_bs_opts opts; + spdk_bs_op_with_handle_complete cb_fn; + void *cb_arg; + struct spdk_bs_super_block *super; + struct spdk_io_channel *ch; + struct spdk_bs_dev_cb_args cb_args; +}; + +static void +bs_load_continue(struct spdk_bs_dev *dev, struct spdk_bs_opts *opts, + spdk_bs_op_with_handle_complete cb_fn, void *cb_arg, bool journaled) +{ + struct spdk_blob_store *bs; + struct spdk_bs_load_ctx *ctx; + struct spdk_bs_md_journal *journal = NULL; + int err; + + if (journaled) { + struct spdk_bs_dev *jdev = bs_md_journal_dev_create(dev, &journal); + + if (jdev == NULL) { + /* the store is flagged as journal-formatted but the + * ring cannot be mapped — loading unprotected would + * read stale metadata */ + SPDK_ERRLOG("store has an md journal but the device cannot carry it\n"); + dev->destroy(dev); + cb_fn(cb_arg, NULL, -EILSEQ); + return; + } + dev = jdev; + } + + err = bs_alloc(dev, opts, &bs, &ctx); + if (err) { + dev->destroy(dev); + cb_fn(cb_arg, NULL, err); + return; + } + bs->md_journal = journal; + + ctx->load_cb_fn = cb_fn; + ctx->load_cb_arg = cb_arg; + + if (bs->md_journal != NULL) { + /* recover the journal (scan ring, rebuild buffer/dictionary) + * before the first md read — reads are overlaid from it */ + bs_md_journal_start(bs->md_journal, false, bs_load_read_super, ctx); + return; + } + bs_load_read_super(ctx, 0); +} + +static void +bs_load_probe_super_cpl(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct bs_load_probe_ctx *probe = cb_arg; + struct spdk_bs_dev *dev = probe->dev; + struct spdk_bs_opts opts = probe->opts; + spdk_bs_op_with_handle_complete cb_fn = probe->cb_fn; + void *probe_cb_arg = probe->cb_arg; + bool journaled; + + journaled = bserrno == 0 && + memcmp(probe->super->signature, SPDK_BS_SUPER_BLOCK_SIG, + sizeof(probe->super->signature)) == 0 && + probe->super->md_journal == 1; + + dev->destroy_channel(dev, probe->ch); + spdk_free(probe->super); + free(probe); + + bs_load_continue(dev, &opts, cb_fn, probe_cb_arg, journaled); +} + void spdk_bs_load(struct spdk_bs_dev *dev, struct spdk_bs_opts *o, spdk_bs_op_with_handle_complete cb_fn, void *cb_arg) { - struct spdk_blob_store *bs; - struct spdk_bs_cpl cpl; - struct spdk_bs_load_ctx *ctx; + struct bs_load_probe_ctx *probe; struct spdk_bs_opts opts = {}; - int err; SPDK_INFOLOG(blob, "Loading blobstore from dev %p\n", dev); @@ -6629,24 +6721,68 @@ spdk_bs_load(struct spdk_bs_dev *dev, struct spdk_bs_opts *o, return; } - err = bs_alloc(dev, &opts, &bs, &ctx); - if (err) { + probe = calloc(1, sizeof(*probe)); + if (probe == NULL) { dev->destroy(dev); - cb_fn(cb_arg, NULL, err); + cb_fn(cb_arg, NULL, -ENOMEM); + return; + } + probe->super = spdk_zmalloc(sizeof(*probe->super), 0x1000, NULL, + SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA); + probe->ch = dev->create_channel(dev); + if (probe->super == NULL || probe->ch == NULL) { + if (probe->ch != NULL) { + dev->destroy_channel(dev, probe->ch); + } + spdk_free(probe->super); + free(probe); + dev->destroy(dev); + cb_fn(cb_arg, NULL, -ENOMEM); + return; + } + probe->dev = dev; + probe->opts = opts; + probe->cb_fn = cb_fn; + probe->cb_arg = cb_arg; + probe->cb_args.cb_fn = bs_load_probe_super_cpl; + probe->cb_args.channel = probe->ch; + probe->cb_args.cb_arg = probe; + + { + struct spdk_bs_io_opts bs_io_opts = {0}; + + dev->read(dev, probe->ch, probe->super, 0, + sizeof(*probe->super) / dev->blocklen, &probe->cb_args, &bs_io_opts); + } +} + +static void +bs_load_read_super(void *cb_arg, int bserrno) +{ + struct spdk_bs_load_ctx *ctx = cb_arg; + struct spdk_blob_store *bs = ctx->bs; + struct spdk_bs_cpl cpl; + + if (bserrno != 0) { + SPDK_ERRLOG("md journal recovery failed: %d\n", bserrno); + spdk_free(ctx->super); + ctx->load_cb_fn(ctx->load_cb_arg, NULL, bserrno); + free(ctx); + bs_free(bs); return; } cpl.type = SPDK_BS_CPL_TYPE_BS_HANDLE; - cpl.u.bs_handle.cb_fn = cb_fn; - cpl.u.bs_handle.cb_arg = cb_arg; + cpl.u.bs_handle.cb_fn = ctx->load_cb_fn; + cpl.u.bs_handle.cb_arg = ctx->load_cb_arg; cpl.u.bs_handle.bs = bs; ctx->seq = bs_sequence_start_bs(bs->md_channel, &cpl); if (!ctx->seq) { spdk_free(ctx->super); + ctx->load_cb_fn(ctx->load_cb_arg, NULL, -ENOMEM); free(ctx); bs_free(bs); - cb_fn(cb_arg, NULL, -ENOMEM); return; } @@ -7371,6 +7507,15 @@ bs_init_persist_super_cpl(spdk_bs_sequence_t *seq, void *cb_arg, int bserrno) { struct spdk_bs_load_ctx *ctx = cb_arg; + if (bserrno == 0 && ctx->bs->md_journal != NULL) { + /* the super block (with the md_journal flag) is home-durable + * now — arm md write/read interception for everything that + * follows */ + bs_md_journal_enable(ctx->bs->md_journal, + bs_page_to_lba(ctx->bs, + ctx->super->md_start + ctx->super->md_len)); + } + ctx->bs->used_clusters = spdk_bit_pool_create_from_array(ctx->used_clusters); spdk_free(ctx->super); free(ctx); @@ -7431,11 +7576,22 @@ spdk_bs_init(struct spdk_bs_dev *dev, struct spdk_bs_opts *o, return; } - rc = bs_alloc(dev, &opts, &bs, &ctx); - if (rc) { - dev->destroy(dev); - cb_fn(cb_arg, NULL, rc); - return; + /* Torn-write protection: reserve the ring by wrapping the device + * before the blobstore sizes itself. */ + { + struct spdk_bs_md_journal *journal = NULL; + struct spdk_bs_dev *jdev = bs_md_journal_dev_create(dev, &journal); + + if (jdev != NULL) { + dev = jdev; + } + rc = bs_alloc(dev, &opts, &bs, &ctx); + if (rc) { + dev->destroy(dev); + cb_fn(cb_arg, NULL, rc); + return; + } + bs->md_journal = journal; } if (opts.num_md_pages == SPDK_BLOB_OPTS_NUM_MD_PAGES) { @@ -7545,6 +7701,16 @@ spdk_bs_init(struct spdk_bs_dev *dev, struct spdk_bs_opts *o, num_md_lba = bs_page_to_lba(bs, num_md_pages); + if (bs->md_journal != NULL) { + /* No recovery on fresh format; the ring itself is zeroed in + * the init batch below. Interception is armed only once the + * super block is home-durable (bs_init_persist_super_cpl): + * the format writes go raw so that a store whose super was + * ever acknowledged always carries the flag on disk. */ + bs_md_journal_start(bs->md_journal, true, NULL, NULL); + ctx->super->md_journal = 1; + } + ctx->super->size = dev->blockcnt * dev->blocklen; ctx->super->crc = blob_md_page_calc_crc(ctx->super); @@ -7598,6 +7764,13 @@ spdk_bs_init(struct spdk_bs_dev *dev, struct spdk_bs_opts *o, // bs->w_io++; // bs_batch_write_zeroes_dev(batch, 0, num_md_lba); + if (bs->md_journal != NULL) { + /* fresh format: zero the journal ring (raw region above the + * proxy's blockcnt; write_zeroes passes through) */ + bs_batch_write_zeroes_dev(batch, bs_md_journal_ring_lba(bs->md_journal), + bs_md_journal_ring_lba_count(bs->md_journal)); + } + lba = num_md_lba; lba_count = ctx->bs->dev->blockcnt - lba; switch (opts.clear_method) { @@ -7976,7 +8149,37 @@ spdk_bs_get_super(struct spdk_blob_store *bs, void spdk_bs_set_leader(struct spdk_blob_store *bs, bool state) { - bs->is_leader = state; + bs->is_leader = state; + if (bs->md_journal != NULL) { + /* only the leader may drain the shared ring - see + * bs_md_journal_set_leader() */ + bs_md_journal_set_leader(bs->md_journal, state); + } +} + +int +spdk_bs_get_md_journal_stats(struct spdk_blob_store *bs, + struct spdk_bs_md_journal_stats *stats) +{ + memset(stats, 0, sizeof(*stats)); + if (bs->md_journal == NULL) { + return -ENODEV; + } + bs_md_journal_get_stats(bs->md_journal, &stats->enabled, &stats->num_slots, + &stats->used_slots, &stats->mem_head, &stats->mem_tail, + &stats->disk_head, &stats->disk_tail, + &stats->drain_paused, &stats->drain_demoted); + return 0; +} + +int +spdk_bs_set_md_journal_drain_paused(struct spdk_blob_store *bs, bool paused) +{ + if (bs->md_journal == NULL) { + return -ENODEV; + } + bs_md_journal_set_drain_paused(bs->md_journal, paused); + return 0; } void @@ -14793,6 +14996,43 @@ bs_update_super_cpl(spdk_bs_sequence_t *seq, void *cb_arg, int bserrno) bs_update_read_only_used_blobid_pages(ctx); } +static void bs_update_live_read_super(struct spdk_bs_update_ctx *ctx); + +/* The ring lives on the shared device, the buffer/dictionary that overlays md + * reads from it does not: it is rebuilt by recovery when the lvstore is + * LOADED. A peer that loaded the lvstore while another node was the leader + * therefore holds a snapshot of the ring as of its own load, and the product + * promotes exactly such a peer (bdev_lvol_update_lvstore + set_leader) rather + * than loading the lvstore anew. Re-read the ring before re-reading md, so a + * takeover sees every acknowledged page the dead leader left undrained and + * appends behind them instead of over them. */ +static void +bs_update_live_journal_rescan_cpl(void *cb_arg, int bserrno) +{ + struct spdk_bs_update_ctx *ctx = cb_arg; + + if (bserrno != 0) { + SPDK_ERRLOG("md journal rescan before md reload failed: %d\n", bserrno); + ctx->bs->r_io--; + bs_update_live_done(ctx, bserrno); + return; + } + ctx->bs->r_io--; + bs_update_live_read_super(ctx); +} + +static void +bs_update_live_read_super(struct spdk_bs_update_ctx *ctx) +{ + struct spdk_blob_store *bs = ctx->bs; + + /* Read the super block */ + bs->r_io++; + bs_sequence_read_dev(ctx->seq, ctx->super, bs_page_to_lba(bs, 0), + bs_byte_to_lba(bs, sizeof(*ctx->super)), + bs_update_super_cpl, ctx); +} + void spdk_bs_update_live(struct spdk_blob_store *bs, bool failover, uint64_t id, spdk_bs_op_complete cb_fn, void *cb_arg) @@ -14834,11 +15074,15 @@ spdk_bs_update_live(struct spdk_blob_store *bs, bool failover, uint64_t id, return; } - /* Read the super block */ - bs->r_io++; - bs_sequence_read_dev(ctx->seq, ctx->super, bs_page_to_lba(bs, 0), - bs_byte_to_lba(bs, sizeof(*ctx->super)), - bs_update_super_cpl, ctx); + if (bs->md_journal != NULL && id == 0) { + /* whole-store reload (promotion / failover), not the per-blob + * variant of bdev_lvol_register: re-read the ring first */ + bs->r_io++; + bs_md_journal_rescan(bs->md_journal, bs_update_live_journal_rescan_cpl, ctx); + return; + } + + bs_update_live_read_super(ctx); } static void diff --git a/lib/blob/blobstore.h b/lib/blob/blobstore.h index 59423c3ba44..6b3b46ec86a 100644 --- a/lib/blob/blobstore.h +++ b/lib/blob/blobstore.h @@ -185,6 +185,9 @@ struct spdk_blob_store { int priority_class; // max priority_class of all constituent blobs to speed up metadata I/Os struct spdk_bs_dev *dev; + /* torn-write-protection journal owning the top of the base dev; + * bs->dev is its proxy when set (see blob_md_journal.h) */ + struct spdk_bs_md_journal *md_journal; struct spdk_bit_array *used_md_pages; /* Protected by used_lock */ struct spdk_bit_pool *used_clusters; /* Protected by used_lock */ @@ -476,7 +479,13 @@ struct spdk_bs_super_block { uint64_t size; /* size of blobstore in bytes */ uint32_t io_unit_size; /* Size of io unit in bytes */ - uint8_t reserved[4000]; + /* Torn-write-protection md journal (blob_md_journal.h): 1 iff the + * store was formatted with the reserved ring region at the top of + * the device. Legacy stores have 0 here and load unchanged. The + * flag never changes over the lifetime of a store. */ + uint32_t md_journal; + + uint8_t reserved[3996]; uint32_t crc; }; SPDK_STATIC_ASSERT(sizeof(struct spdk_bs_super_block) == 0x1000, "Invalid super block size"); diff --git a/lib/blob/spdk_blob.map b/lib/blob/spdk_blob.map index 14c105f7c3a..c870efdd0eb 100644 --- a/lib/blob/spdk_blob.map +++ b/lib/blob/spdk_blob.map @@ -11,6 +11,8 @@ spdk_bs_set_super; spdk_bs_get_super; spdk_bs_get_cluster_size; + spdk_bs_get_md_journal_stats; + spdk_bs_set_md_journal_drain_paused; spdk_bs_get_page_size; spdk_bs_get_io_unit_size; spdk_bs_free_cluster_count; diff --git a/md_journal_test_plan.md b/md_journal_test_plan.md new file mode 100644 index 00000000000..5906587ee81 --- /dev/null +++ b/md_journal_test_plan.md @@ -0,0 +1,89 @@ +# md-journal test plan (branch md-journal) + +## 1. C unit tests (SPDK CUnit, test/unit/lib/blob/) +New suite `test/unit/lib/blob/blob_md_journal.c/blob_md_journal_ut.c` with an +in-memory mock bs_dev (pattern: existing blob ut bs_dev in test/unit/lib/blob). +Cases: +- U1 append/ack ordering: caller cb only after ring write; entry bytes on "disk" +- U2 read overlay: journaled-not-drained page served from dict, home stale +- U3 drain: home updated, both entry blocks zeroed, dict entry dropped +- U4 supersede-coalescing: two appends same LBA -> older drained without home write +- U5 ring full: 8191 entries stall the next append; drain frees -> proceeds +- U6 recovery: valid contiguous run rebuilt (tail/head/dict), drains after +- U7 recovery with torn entry (bad crc) and zeroed entries -> treated empty +- U8 recovery duplicates: later ring position wins in dict +- U9 power-off simulation: snapshot mock-dev buffer mid-workload, new journal + instance on snapshot -> every acked write recoverable, unacked may vanish +- U10 proxy geometry: blockcnt shrunk by 64 MB; data write above limit passthrough +- U11 multi-page append (mask-style N-page write) FIFO order + single ack +- U12 destroy with in-flight IO (deferred teardown, no use-after-free) +- U13 read-vs-drain race: home read in flight while drain completes -> + issue-time dict snapshot must win over stale device data (found + as blob-md crc mismatch in single-node integration, 2026-08-04) + +## 2. Build (EC2 Rocky 9 / Fedora) +Per ultra CI (docker/Dockerfile_spdk_ultra): spdk configured+built, then ultra +(CMake, DISTR_v2) links spdk. For unit tests only: spdk ./configure +--with-fio... not needed; `./configure && make -j` + `./test/unit/unittest.sh` +or direct CUnit binary. Host: i3en/m6i Rocky 9 (AMI ami-0dfc569a8686b9320, +key mtes01) — same constants as sbcli/scripts/setup_perf_test_multipath.py. + +## 3. Integration tests (ultra python style, single node) +Model on ultra/testing + scripts/run_lvs_tests.sh (CI: spdk_container_lvstore_ +unit_tests.yml — container + python suite driving RPCs): +- I1 regular ops: create lvstore/lvols/snapshots under IO; md reads/writes + correct while journal active (lvol list/get consistency, blob md intact + after bdev_lvol_* cycles) +- I2 sudden power-off: kill -9 SPDK container mid md-heavy workload + (create/delete loop); restart; lvstore must load, all acked objects + present, no torn-md load abort; repeat N rounds +- I3 torn-write injection: with ALLOW_FAILURE_GENERATION build flag (CI uses + it), or dd partial 512B overwrite of a ring entry + of a home md page + while down -> load must recover/ignore correctly + NOTE representativeness: neither kill -9 (io_submit'ed IOs complete + in-kernel, unsubmitted ones never start) nor AWS Nitro storage (16 KiB + torn-write prevention on EBS/instance store) produces naturally torn + 4K writes, so ALL torn states must be synthesized by injection; the + injected states model the 512B-sector-atomic worst case of the distr + virtual device (the design's actual target) +- I6 torn HOME page repair (the headline scenario): synthesize valid ring + entries (bit-exact magic+crc32c) for real md pages, tear those home + pages (and the super, keeping sector 0) on the raw file -> load must + succeed and repair every page byte-exactly from the ring +- I4 journal-full pressure: mass create/delete driving ring saturation; + operations stall but complete; no errors +- I5 restart-recovery drain: fill ring, power off, restart, verify ring + drains to empty (entries zeroed) and md pages land home + +## 4. Failover — DONE (ultra scripts/run_mdj_failover_tests.sh, +## DISTR_v2/src_scripts_test_local/mdj_failover_tests.py) +Rig: three bdts processes on one host. A device server owns the NVMe, builds +alceml (+jm) -> 2 distribs -> raid0 and exports the raid over nvmf-tcp; nodes +A and B (own RPC socket / shm-id / hugepages, --no-pci) both attach that one +namespace, so the lvstore's backing virtual device is shared exactly as a +primary/secondary pair shares a distr. The device server doubles as an +out-of-band ring observer (dbg_direct_io_read on the raid), so ring state is +read without asking either lvstore node. + +- F1 failover integrity, >= 4 rounds alternating roles and takeover mode: + 'load' (peer bdev_examine's after the leader dies — design §8) and + 'preloaded' (peer loaded the lvstore while the leader was alive and is + promoted with bdev_lvol_update_lvstore + bdev_lvol_set_leader_all — the + product path). Every acked create present, every sync-acked delete gone. +- F2 recovery with entries: the drain keeps the ring at ~1 entry under any + workload (6 parallel md clients still left 0 at kill time), so the + journal exposes bdev_lvol_set_md_journal_drain for tests: pause, build a + backlog of acked-but-not-home pages, kill -9, and require the takeover + to replay exactly that many entries and drain to empty. +- F3 single-writer fencing (§11.2): SIGSTOP the leader, promote the peer, + SIGCONT the old leader and let it attempt md writes. +- F4 journal-full failover: same as F2 with a deep backlog. + +## Status / blockers +- Phases 1-3 green. Phase 3 produced one fix (journal rescan on promotion, + design §8.1) and the drain-pause / ring-stats test interface + (bdev_lvol_get_md_journal_stats, bdev_lvol_set_md_journal_drain). +- Not covered by the single-host rig: real multi-node behaviour on an + sbcli-deployed cluster (hublvol redirect IO, ANA multipath, CP-driven + demote/promote ordering). Those need the multipath harness on a real + 2-3 node cluster. diff --git a/module/bdev/lvol/vbdev_lvol_rpc.c b/module/bdev/lvol/vbdev_lvol_rpc.c index 64f8aed5746..1d961cf7f3e 100644 --- a/module/bdev/lvol/vbdev_lvol_rpc.c +++ b/module/bdev/lvol/vbdev_lvol_rpc.c @@ -2756,6 +2756,119 @@ rpc_bdev_lvol_block_data_port(struct spdk_jsonrpc_request *request, SPDK_RPC_REGISTER("bdev_lvol_block_data_port", rpc_bdev_lvol_block_data_port, SPDK_RPC_RUNTIME) +/* md journal (blob_md_journal.h): ring introspection and the drain test hook. + * Both are per-LVS and act on the blobstore of this process only - the ring + * itself is shared with the peers on the same virtual device, the in-memory + * view reported here is not. */ +struct rpc_bdev_lvol_md_journal { + char *uuid; + char *lvs_name; + bool paused; +}; + +static void +free_rpc_bdev_lvol_md_journal(struct rpc_bdev_lvol_md_journal *req) +{ + free(req->uuid); + free(req->lvs_name); +} + +static const struct spdk_json_object_decoder rpc_bdev_lvol_md_journal_decoders[] = { + {"uuid", offsetof(struct rpc_bdev_lvol_md_journal, uuid), spdk_json_decode_string, true}, + {"lvs_name", offsetof(struct rpc_bdev_lvol_md_journal, lvs_name), spdk_json_decode_string, true}, +}; + +static const struct spdk_json_object_decoder rpc_bdev_lvol_md_journal_drain_decoders[] = { + {"uuid", offsetof(struct rpc_bdev_lvol_md_journal, uuid), spdk_json_decode_string, true}, + {"lvs_name", offsetof(struct rpc_bdev_lvol_md_journal, lvs_name), spdk_json_decode_string, true}, + {"paused", offsetof(struct rpc_bdev_lvol_md_journal, paused), spdk_json_decode_bool}, +}; + +static void +rpc_bdev_lvol_get_md_journal_stats(struct spdk_jsonrpc_request *request, + const struct spdk_json_val *params) +{ + struct rpc_bdev_lvol_md_journal req = {}; + struct spdk_bs_md_journal_stats stats = {}; + struct spdk_lvol_store *lvs = NULL; + struct spdk_json_write_ctx *w; + int rc; + + if (spdk_json_decode_object(params, rpc_bdev_lvol_md_journal_decoders, + SPDK_COUNTOF(rpc_bdev_lvol_md_journal_decoders), + &req)) { + spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR, + "spdk_json_decode_object failed"); + goto cleanup; + } + + rc = vbdev_get_lvol_store_by_uuid_xor_name(req.uuid, req.lvs_name, &lvs); + if (rc != 0) { + spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc)); + goto cleanup; + } + + rc = spdk_bs_get_md_journal_stats(lvs->blobstore, &stats); + if (rc != 0) { + spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc)); + goto cleanup; + } + + w = spdk_jsonrpc_begin_result(request); + spdk_json_write_object_begin(w); + spdk_json_write_named_bool(w, "enabled", stats.enabled); + spdk_json_write_named_bool(w, "drain_paused", stats.drain_paused); + spdk_json_write_named_bool(w, "drain_demoted", stats.drain_demoted); + spdk_json_write_named_uint32(w, "num_slots", stats.num_slots); + spdk_json_write_named_uint32(w, "used_slots", stats.used_slots); + spdk_json_write_named_uint32(w, "mem_head", stats.mem_head); + spdk_json_write_named_uint32(w, "mem_tail", stats.mem_tail); + spdk_json_write_named_uint32(w, "disk_head", stats.disk_head); + spdk_json_write_named_uint32(w, "disk_tail", stats.disk_tail); + spdk_json_write_object_end(w); + spdk_jsonrpc_end_result(request, w); + +cleanup: + free_rpc_bdev_lvol_md_journal(&req); +} +SPDK_RPC_REGISTER("bdev_lvol_get_md_journal_stats", rpc_bdev_lvol_get_md_journal_stats, + SPDK_RPC_RUNTIME) + +static void +rpc_bdev_lvol_set_md_journal_drain(struct spdk_jsonrpc_request *request, + const struct spdk_json_val *params) +{ + struct rpc_bdev_lvol_md_journal req = {}; + struct spdk_lvol_store *lvs = NULL; + int rc; + + if (spdk_json_decode_object(params, rpc_bdev_lvol_md_journal_drain_decoders, + SPDK_COUNTOF(rpc_bdev_lvol_md_journal_drain_decoders), + &req)) { + spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INTERNAL_ERROR, + "spdk_json_decode_object failed"); + goto cleanup; + } + + rc = vbdev_get_lvol_store_by_uuid_xor_name(req.uuid, req.lvs_name, &lvs); + if (rc != 0) { + spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc)); + goto cleanup; + } + + rc = spdk_bs_set_md_journal_drain_paused(lvs->blobstore, req.paused); + if (rc != 0) { + spdk_jsonrpc_send_error_response(request, rc, spdk_strerror(-rc)); + goto cleanup; + } + spdk_jsonrpc_send_bool_response(request, true); + +cleanup: + free_rpc_bdev_lvol_md_journal(&req); +} +SPDK_RPC_REGISTER("bdev_lvol_set_md_journal_drain", rpc_bdev_lvol_set_md_journal_drain, + SPDK_RPC_RUNTIME) + struct rpc_bdev_lvol_shallow_copy { char *src_lvol_name; char *dst_bdev_name; diff --git a/test/md_journal/mdj_integration.py b/test/md_journal/mdj_integration.py new file mode 100644 index 00000000000..cf381c962e8 --- /dev/null +++ b/test/md_journal/mdj_integration.py @@ -0,0 +1,530 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (C) 2026 Simplyblock GmbH. +# +# Single-node integration tests for the blobstore md journal +# (lib/blob/blob_md_journal.c), cases I1-I5 of md_journal_test_plan.md, +# without the ultra stack: a real spdk_tgt with an lvstore on a +# file-backed AIO bdev (large enough that the journal activates), +# driven over RPC. +# +# I1 regular ops: lvol/snapshot lifecycle under an active journal +# I2 sudden power-off: kill -9 mid create/delete workload, restart, +# every RPC-acked object present, every acked delete gone, no +# torn-md load abort; repeated rounds +# I3 torn-write injection: garbage / fake-header 512B sectors written +# into empty ring slots while down -> load recovers/ignores +# I4 journal pressure: parallel mass create/delete, all ops succeed +# I5 restart-recovery drain: ring drains to all-zero after restart, +# objects intact +# +# Run as root on the build host: sudo python3 mdj_integration.py + +import json +import os +import random +import signal +import struct +import subprocess +import sys +import threading +import time + +SPDK_DIR = os.environ.get("SPDK_DIR", "/home/ec2-user/spdk") +SPDK_TGT = f"{SPDK_DIR}/build/bin/spdk_tgt" +RPC_PY = f"{SPDK_DIR}/scripts/rpc.py" +SOCK = "/var/tmp/mdj_it.sock" +TGT_LOG = "/tmp/mdj_tgt.log" +IMG = os.environ.get("MDJ_IMG", "/home/ec2-user/mdj_aio.img") +IMG_SIZE = 4 * 1024 ** 3 +BLOCKLEN = 4096 +RING_BYTES = 64 * 1024 ** 2 +RING_START = IMG_SIZE - RING_BYTES +ENTRY_BYTES = 8192 +NUM_SLOTS = RING_BYTES // ENTRY_BYTES +HDR_MAGIC = 0x4D444A31 +LVS = "lvs0" + +g_tgt = None +g_failures = [] + + +def fail(msg): + print(f" FAIL: {msg}") + g_failures.append(msg) + + +def check(cond, msg): + if cond: + return True + fail(msg) + return False + + +# ------------------------------------------------------------------ # +# target / rpc plumbing # + +def rpc_raw(*args, timeout=90): + return subprocess.run( + [sys.executable, RPC_PY, "-s", SOCK, "-t", str(timeout)] + [str(a) for a in args], + capture_output=True, text=True) + + +def rpc(*args): + p = rpc_raw(*args) + if p.returncode != 0: + raise RuntimeError(f"rpc {args[0]} failed: {p.stderr.strip()[:300]}") + return p.stdout + + +def rpc_json(*args): + return json.loads(rpc(*args)) + + +def start_tgt(): + global g_tgt + logf = open(TGT_LOG, "a") + logf.write(f"\n===== tgt start {time.ctime()} =====\n") + logf.flush() + # --disable-cpumask-locks: kill -9 rounds leave stale core-lock files + g_tgt = subprocess.Popen([SPDK_TGT, "-r", SOCK, "-m", "0x3", "-s", "1024", + "--disable-cpumask-locks"], + stdout=logf, stderr=logf) + deadline = time.time() + 60 + while time.time() < deadline: + if rpc_raw("rpc_get_methods", timeout=2).returncode == 0: + return + if g_tgt.poll() is not None: + raise RuntimeError(f"spdk_tgt exited at startup, rc={g_tgt.returncode}, see {TGT_LOG}") + time.sleep(0.25) + raise RuntimeError("spdk_tgt did not come up") + + +def kill9_tgt(): + g_tgt.send_signal(signal.SIGKILL) + g_tgt.wait() + + +def stop_tgt_clean(): + g_tgt.send_signal(signal.SIGTERM) + try: + g_tgt.wait(timeout=60) + except subprocess.TimeoutExpired: + g_tgt.kill() + g_tgt.wait() + + +def attach_and_load(): + """Create the AIO bdev; lvol examine loads the lvstore (journal + recovery runs inside spdk_bs_load). Returns the loaded lvstore.""" + rpc("bdev_aio_create", IMG, "aio0", BLOCKLEN) + deadline = time.time() + 120 + while time.time() < deadline: + stores = rpc_json("bdev_lvol_get_lvstores") + if stores and stores[0]["name"] == LVS: + # let examine finish registering all lvol bdevs + prev = -1 + while time.time() < deadline: + cur = len(present_lvols()) + if cur == prev: + return stores[0] + prev = cur + time.sleep(0.5) + time.sleep(0.25) + raise RuntimeError("lvstore did not load within 120s (torn-md load abort?)") + + +def present_lvols(): + """Set of 'name' for every lvol/snapshot bdev of LVS.""" + out = set() + for b in rpc_json("bdev_get_bdevs"): + for a in b.get("aliases", []): + if a.startswith(f"{LVS}/"): + out.add(a.split("/", 1)[1]) + return out + + +def journal_active_in_log(): + with open(TGT_LOG) as f: + return "md journal enabled" in f.read() + + +# ------------------------------------------------------------------ # +# ring inspection on the raw backing file (target must be down) # + +def ring_slots(): + """Return (valid_slots, nonzero_slots) parsed from the ring. + The target writes with O_DIRECT, our reads are buffered: drop the + page cache first so we see the platter state.""" + valid, nonzero = [], [] + with open("/proc/sys/vm/drop_caches", "w") as f: + f.write("1") + with open(IMG, "rb") as f: + f.seek(RING_START) + ring = f.read(RING_BYTES) + for s in range(NUM_SLOTS): + entry = ring[s * ENTRY_BYTES:(s + 1) * ENTRY_BYTES] + if any(entry): + nonzero.append(s) + magic = struct.unpack_from(" + # crc mismatch -> must be treated as empty + with open(IMG, "rb") as f: + if valid: + f.seek(RING_START + valid[0] * ENTRY_BYTES) + fake_hdr = f.read(512) + else: + fake_hdr = struct.pack(" + # a read-only look at the backing file is stable + valid, nonzero = ring_slots() + # inert garbage injected by I3 far from the head is empty by + # definition and only reclaimed when the head wraps over it + leftover = [s for s in nonzero if s not in g_injected_slots] + if not valid and not leftover: + break + check(valid == [], f"I5: {len(valid or [])} valid entries never drained") + check(leftover == [], f"I5: {len(leftover or [])} unexplained non-zero slots after idle drain") + check(present_lvols() == before, "I5: objects changed while draining") + # a final power-off on the fully-drained ring must recover to the + # same object set (empty-ring recovery path) + kill9_tgt() + start_tgt() + attach_and_load() + check(present_lvols() == before, "I5: objects differ after empty-ring recovery") + check(verify_after_restart(all_workloads, "I5"), "I5: final object set wrong") + print(f" ok ({len(before)} objects, all journal entries drained+zeroed)") + + +def crc32c_raw(data): + """spdk_crc32c_update(buf, len, 0): raw reflected CRC32C (Castagnoli), + init as passed (0), no final inversion.""" + crc = 0 + for b in data: + crc ^= b + for _ in range(8): + crc = (crc >> 1) ^ (0x82F63B78 if crc & 1 else 0) + return crc + + +def t6_torn_home_repair(all_workloads): + """I6: THE torn-write-protection scenario, synthesized deterministically. + + Neither kill -9 (io_submit'ed IOs complete in-kernel; unsubmitted ones + never start) nor AWS Nitro storage (16 KiB torn-write prevention on + EBS/instance store) can produce naturally torn 4K writes on this rig, + so the power-loss state of 512B-atomic storage is synthesized: valid + ring entries holding the newest md pages, with the corresponding HOME + pages torn mid-write. Load must repair every one of them byte-exactly + from the ring.""" + print("I6 torn home md pages repaired from the ring") + # quiesce with a drained ring (idle store drains fast); inert garbage + # slots injected by I3 are empty by definition and stay until the + # head wraps over them + valid, leftover = None, None + for _ in range(3): + kill9_tgt() + valid, nonzero = ring_slots() + leftover = [s for s in nonzero if s not in g_injected_slots] + if not valid and not leftover: + break + start_tgt() + attach_and_load() + time.sleep(5) + if not check(valid == [] and leftover == [], + f"I6: could not reach a drained ring (valid={valid} other={leftover})"): + return + + with open(IMG, "rb") as f: + img_read = f.read(1028 * BLOCKLEN) + + def page(lba): + return img_read[lba * BLOCKLEN:(lba + 1) * BLOCKLEN] + + # three blob md pages (deep in the md region: stable during idle load) + victims = [l for l in range(100, 1028) if any(page(l))][:3] + check(len(victims) == 3, f"I6: found only {len(victims)} blob md pages to tear") + saved = {l: page(l) for l in victims} + super_page = page(0) + + with open(IMG, "r+b") as f: + # synthetic ring entries (slots 0..3): [header][newest page] + for slot, lba in enumerate(victims + [0]): + content = saved.get(lba, super_page) + hdr = struct.pack(" load abort + check(verify_after_restart(all_workloads, "I6"), "I6: object set wrong after torn-home repair") + + # the drain must land the ring copies home, byte-exact + deadline = time.time() + 60 + while time.time() < deadline: + time.sleep(3) + valid, _ = ring_slots() + if not valid: + break + check(valid == [], "I6: synthetic entries never drained") + with open(IMG, "rb") as f: + for lba in victims: + f.seek(lba * BLOCKLEN) + repaired = f.read(BLOCKLEN) + check(repaired == saved[lba], f"I6: home page lba {lba} not repaired byte-exactly") + + with open(TGT_LOG) as f: + f.seek(log_mark) + boot_log = f.read() + check("crc mismatch" not in boot_log, "I6: md crc errors during repaired load") + check("Metadata page" not in boot_log, "I6: blob md errors during repaired load") + print(f" ok (3 torn blob md pages + torn super repaired from ring, lbas {victims})") + + +def main(): + random.seed(20260804) + if os.geteuid() != 0: + print("must run as root (hugepages/spdk_tgt)") + return 1 + subprocess.run(["pkill", "-9", "-x", "spdk_tgt"], capture_output=True) + if os.path.exists(IMG): + os.unlink(IMG) + with open(IMG, "wb") as f: + f.truncate(IMG_SIZE) + + start_tgt() + rpc("bdev_aio_create", IMG, "aio0", BLOCKLEN) + + t1_regular_ops() + workloads = t2_kill9_rounds() + t3_torn_injection(workloads) + t4_mass_pressure() + t5_ring_drain(workloads) + t6_torn_home_repair(workloads) + + stop_tgt_clean() + print() + if g_failures: + print(f"RESULT: {len(g_failures)} FAILURE(S)") + for m in g_failures: + print(f" - {m}") + return 1 + print("RESULT: all integration tests passed (I1-I6)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/unit/lib/blob/Makefile b/test/unit/lib/blob/Makefile index 01800a506d2..04c4a2b792e 100644 --- a/test/unit/lib/blob/Makefile +++ b/test/unit/lib/blob/Makefile @@ -11,8 +11,8 @@ include $(SPDK_ROOT_DIR)/mk/spdk.common.mk # rather than on configuration values. All sub-directories are # added to $(DIRS-y) so that they are included in 'make clean'. # $(ALL_DIRS) contains the list of sub-directories to compile. -DIRS-y = blob.c blob_bdev.c -ALL_DIRS = blob_bdev.c +DIRS-y = blob.c blob_bdev.c blob_md_journal.c +ALL_DIRS = blob_bdev.c blob_md_journal.c HASH = \# CUNIT_VERSION = $(shell echo "$(HASH)include " | $(CC) $(CFLAGS) -E -dM - | sed -n -e 's/\#define CU_VERSION "\([0-9\.\-]*\).*/\1/p') diff --git a/test/unit/lib/blob/blob.c/blob_ut.c b/test/unit/lib/blob/blob.c/blob_ut.c index a26b3e47688..778026a2ea3 100644 --- a/test/unit/lib/blob/blob.c/blob_ut.c +++ b/test/unit/lib/blob/blob.c/blob_ut.c @@ -15,6 +15,7 @@ #include "thread/thread.c" #include "ext_dev.c" #include "blob/blobstore.c" +#include "blob/blob_md_journal.c" #include "blob/request.c" #include "blob/zeroes.c" #include "blob/blob_bs_dev.c" diff --git a/test/unit/lib/blob/blob_md_journal.c/Makefile b/test/unit/lib/blob/blob_md_journal.c/Makefile new file mode 100644 index 00000000000..bb37d49757c --- /dev/null +++ b/test/unit/lib/blob/blob_md_journal.c/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (C) 2026 Simplyblock GmbH. +# + +SPDK_ROOT_DIR := $(abspath $(CURDIR)/../../../../..) + +TEST_FILE = blob_md_journal_ut.c + +include $(SPDK_ROOT_DIR)/mk/spdk.unittest.mk diff --git a/test/unit/lib/blob/blob_md_journal.c/blob_md_journal_ut.c b/test/unit/lib/blob/blob_md_journal.c/blob_md_journal_ut.c new file mode 100644 index 00000000000..87fffcd4eb9 --- /dev/null +++ b/test/unit/lib/blob/blob_md_journal.c/blob_md_journal_ut.c @@ -0,0 +1,1222 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (C) 2026 Simplyblock GmbH. + * + * Unit tests for the LVS metadata (page) journal with torn-write + * protection (lib/blob/blob_md_journal.c). Cases U1-U12 from + * md_journal_test_plan.md. + * + * The tests drive the journal through its proxy bs_dev directly (no + * full blobstore) against an in-memory mock bs_dev with fully manual + * IO completion, so that ack ordering (I1), drain ordering (I2) and + * FIFO order (I3) can be asserted at every step. + */ + +#include "spdk/stdinc.h" + +#include "spdk_internal/cunit.h" +#include "spdk/blob.h" + +#include "thread/thread_internal.h" + +#include "common/lib/ut_multithread.c" +#include "thread/thread.c" +#include "blob/blob_md_journal.c" + +/* default routing opts for all UT-issued proxy IO (fork bs_io_opts) */ +static struct spdk_bs_io_opts g_ut_io_opts; + +#define UT_BLOCKLEN 4096 +#define UT_DEV_SIZE (192ULL * 1024 * 1024) +#define UT_BLOCKCNT (UT_DEV_SIZE / UT_BLOCKLEN) +#define UT_JOURNAL_BLOCKS (BS_MD_JOURNAL_SIZE_BYTES / UT_BLOCKLEN) +#define UT_MD_LIMIT_LBA 16384 /* first 64 MB are "metadata" */ + +/* ------------------------------------------------------------------ */ +/* mock bs_dev: flat buffer, manual completion queue */ + +enum ut_io_type { + UT_IO_READ, + UT_IO_WRITE, + UT_IO_WRITE_ZEROES, +}; + +struct ut_io { + enum ut_io_type type; + uint64_t lba; + uint32_t lba_count; + uint8_t *wdata; /* write payload copy, applied on completion */ + void *rpayload; /* read destination (payload form) */ + struct iovec *riov; /* read destination (iov form) */ + int riovcnt; + struct spdk_bs_dev_cb_args *cb_args; + TAILQ_ENTRY(ut_io) link; +}; + +struct ut_dev { + struct spdk_bs_dev bs_dev; + uint8_t *buf; /* not owned */ + bool destroyed; + uint32_t pending; + uint64_t writes_completed[3]; /* per ut_io_type counter */ + TAILQ_HEAD(, ut_io) io_queue; +}; + +struct spdk_io_channel g_ut_io_channel; + +static struct spdk_io_channel * +ut_dev_create_channel(struct spdk_bs_dev *dev) +{ + return &g_ut_io_channel; +} + +static void +ut_dev_destroy_channel(struct spdk_bs_dev *dev, struct spdk_io_channel *channel) +{ +} + +static void +ut_dev_destroy(struct spdk_bs_dev *dev) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + + CU_ASSERT(d->pending == 0); + d->destroyed = true; +} + +static struct ut_io * +ut_io_alloc(struct ut_dev *d, enum ut_io_type type, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args) +{ + struct ut_io *io = calloc(1, sizeof(*io)); + + SPDK_CU_ASSERT_FATAL(io != NULL); + io->type = type; + io->lba = lba; + io->lba_count = lba_count; + io->cb_args = cb_args; + TAILQ_INSERT_TAIL(&d->io_queue, io, link); + d->pending++; + return io; +} + +/* Reads snapshot the device content at ISSUE time (worst-case device + * behavior: a read overlapping a concurrent write may return the old + * data) — this is what exposes read-vs-drain ordering bugs. */ +static void +ut_dev_read_snapshot(struct ut_dev *d, struct ut_io *io) +{ + uint64_t off = io->lba * UT_BLOCKLEN; + uint64_t len = (uint64_t)io->lba_count * UT_BLOCKLEN; + + SPDK_CU_ASSERT_FATAL(off + len <= UT_DEV_SIZE); + io->wdata = malloc(len); + SPDK_CU_ASSERT_FATAL(io->wdata != NULL); + memcpy(io->wdata, d->buf + off, len); +} + +static void +ut_dev_read(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, void *payload, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + struct ut_io *io = ut_io_alloc(d, UT_IO_READ, lba, lba_count, cb_args); + + io->rpayload = payload; + ut_dev_read_snapshot(d, io); +} + +static void +ut_dev_readv(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + struct ut_io *io = ut_io_alloc(d, UT_IO_READ, lba, lba_count, cb_args); + + io->riov = iov; + io->riovcnt = iovcnt; + ut_dev_read_snapshot(d, io); +} + +static void +ut_dev_write(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, void *payload, + uint64_t lba, uint32_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + struct ut_io *io = ut_io_alloc(d, UT_IO_WRITE, lba, lba_count, cb_args); + uint64_t len = (uint64_t)lba_count * UT_BLOCKLEN; + + io->wdata = malloc(len); + SPDK_CU_ASSERT_FATAL(io->wdata != NULL); + memcpy(io->wdata, payload, len); +} + +static void +ut_dev_writev(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct iovec *iov, int iovcnt, uint64_t lba, uint32_t lba_count, + struct spdk_bs_dev_cb_args *cb_args, struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + struct ut_io *io = ut_io_alloc(d, UT_IO_WRITE, lba, lba_count, cb_args); + uint64_t len = (uint64_t)lba_count * UT_BLOCKLEN; + uint64_t off = 0; + int i; + + io->wdata = malloc(len); + SPDK_CU_ASSERT_FATAL(io->wdata != NULL); + for (i = 0; i < iovcnt; i++) { + memcpy(io->wdata + off, iov[i].iov_base, iov[i].iov_len); + off += iov[i].iov_len; + } + CU_ASSERT(off == len); +} + +static void +ut_dev_write_zeroes(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + uint64_t lba, uint64_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + + ut_io_alloc(d, UT_IO_WRITE_ZEROES, lba, lba_count, cb_args); +} + +static void +ut_dev_flush(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + struct spdk_bs_dev_cb_args *cb_args) +{ + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, 0); +} + +static void +ut_dev_unmap(struct spdk_bs_dev *dev, struct spdk_io_channel *channel, + uint64_t lba, uint64_t lba_count, struct spdk_bs_dev_cb_args *cb_args, + struct spdk_bs_io_opts *bs_io_opts) +{ + struct ut_dev *d = SPDK_CONTAINEROF(dev, struct ut_dev, bs_dev); + + ut_io_alloc(d, UT_IO_WRITE_ZEROES, lba, lba_count, cb_args); +} + +/* Complete the oldest pending IO: apply its effect to the flat buffer, + * then run the caller's completion (inline). Returns false when idle. */ +static bool +ut_dev_complete_one(struct ut_dev *d) +{ + struct ut_io *io = TAILQ_FIRST(&d->io_queue); + struct spdk_bs_dev_cb_args *cb_args; + uint64_t off, len; + int i; + + if (io == NULL) { + return false; + } + TAILQ_REMOVE(&d->io_queue, io, link); + d->pending--; + + off = io->lba * UT_BLOCKLEN; + len = (uint64_t)io->lba_count * UT_BLOCKLEN; + SPDK_CU_ASSERT_FATAL(off + len <= UT_DEV_SIZE); + + switch (io->type) { + case UT_IO_READ: + /* serve the issue-time snapshot, not the current content */ + if (io->rpayload != NULL) { + memcpy(io->rpayload, io->wdata, len); + } else { + uint64_t pos = 0; + + for (i = 0; i < io->riovcnt; i++) { + memcpy(io->riov[i].iov_base, io->wdata + pos, io->riov[i].iov_len); + pos += io->riov[i].iov_len; + } + } + break; + case UT_IO_WRITE: + memcpy(d->buf + off, io->wdata, len); + break; + case UT_IO_WRITE_ZEROES: + memset(d->buf + off, 0, len); + break; + } + d->writes_completed[io->type]++; + + cb_args = io->cb_args; + free(io->wdata); + free(io); + cb_args->cb_fn(cb_args->channel, cb_args->cb_arg, 0); + return true; +} + +static uint32_t +ut_dev_complete_all(struct ut_dev *d) +{ + uint32_t n = 0; + + while (ut_dev_complete_one(d)) { + n++; + } + return n; +} + +/* complete the oldest pending IO of @type out of order (device-level + * reordering of concurrent IOs) */ +static bool +ut_dev_complete_first_of(struct ut_dev *d, enum ut_io_type type) +{ + struct ut_io *io; + + TAILQ_FOREACH(io, &d->io_queue, link) { + if (io->type == type) { + /* move to the head, then complete it */ + TAILQ_REMOVE(&d->io_queue, io, link); + TAILQ_INSERT_HEAD(&d->io_queue, io, link); + return ut_dev_complete_one(d); + } + } + return false; +} + +/* Alternate pollers (drain) and IO completion until the system is idle. */ +static void +ut_settle(struct ut_dev *d) +{ + bool progress = true; + + while (progress) { + progress = false; + poll_threads(); + if (ut_dev_complete_all(d) > 0) { + progress = true; + } + } +} + +static void +ut_dev_init(struct ut_dev *d, uint8_t *buf) +{ + memset(d, 0, sizeof(*d)); + d->buf = buf; + TAILQ_INIT(&d->io_queue); + d->bs_dev.blockcnt = UT_BLOCKCNT; + d->bs_dev.blocklen = UT_BLOCKLEN; + d->bs_dev.create_channel = ut_dev_create_channel; + d->bs_dev.destroy_channel = ut_dev_destroy_channel; + d->bs_dev.destroy = ut_dev_destroy; + d->bs_dev.read = ut_dev_read; + d->bs_dev.write = ut_dev_write; + d->bs_dev.readv = ut_dev_readv; + d->bs_dev.writev = ut_dev_writev; + d->bs_dev.flush = ut_dev_flush; + d->bs_dev.write_zeroes = ut_dev_write_zeroes; + d->bs_dev.unmap = ut_dev_unmap; +} + +/* ------------------------------------------------------------------ */ +/* helpers */ + +struct ut_cb_ctx { + bool done; + int rc; +}; + +static void +ut_io_cb(struct spdk_io_channel *ch, void *cb_arg, int bserrno) +{ + struct ut_cb_ctx *ctx = cb_arg; + + CU_ASSERT(!ctx->done); + ctx->done = true; + ctx->rc = bserrno; +} + +static void +ut_start_cb(void *cb_arg, int bserrno) +{ + struct ut_cb_ctx *ctx = cb_arg; + + CU_ASSERT(!ctx->done); + ctx->done = true; + ctx->rc = bserrno; +} + +static void +ut_fill_page(uint8_t *page, uint64_t lba, uint8_t seed) +{ + uint32_t i; + + for (i = 0; i < BS_MD_JOURNAL_PAGE_SIZE; i++) { + page[i] = (uint8_t)(seed ^ (lba & 0xff) ^ (i & 0xff)); + } +} + +static struct ut_dev g_base; +static uint8_t *g_buf; +static struct spdk_bs_md_journal *g_jr; +static struct spdk_bs_dev *g_proxy; + +/* create + start(fresh) + enable a journal over a zeroed mock dev */ +static void +ut_journal_setup(void) +{ + struct ut_cb_ctx start_ctx = {}; + + g_buf = calloc(1, UT_DEV_SIZE); + SPDK_CU_ASSERT_FATAL(g_buf != NULL); + ut_dev_init(&g_base, g_buf); + + g_jr = NULL; + g_proxy = bs_md_journal_dev_create(&g_base.bs_dev, &g_jr); + SPDK_CU_ASSERT_FATAL(g_proxy != NULL); + SPDK_CU_ASSERT_FATAL(g_jr != NULL); + + /* fresh format: ring region already zeroed (calloc); completes inline */ + bs_md_journal_start(g_jr, true, ut_start_cb, &start_ctx); + CU_ASSERT(start_ctx.done && start_ctx.rc == 0); + + bs_md_journal_enable(g_jr, UT_MD_LIMIT_LBA); +} + +static void +ut_journal_teardown(void) +{ + ut_settle(&g_base); + g_proxy->destroy(g_proxy); + CU_ASSERT(g_base.destroyed); + free(g_buf); + g_buf = NULL; + g_jr = NULL; + g_proxy = NULL; +} + +/* append one 4K md page and complete its journal write */ +static void +ut_append_page(uint64_t lba, uint8_t seed) +{ + struct ut_cb_ctx ctx = {}; + uint8_t *page = malloc(BS_MD_JOURNAL_PAGE_SIZE); + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + + SPDK_CU_ASSERT_FATAL(page != NULL); + ut_fill_page(page, lba, seed); + g_proxy->write(g_proxy, NULL, page, lba, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(!ctx.done); /* I1: no ack before journal write */ + CU_ASSERT(ut_dev_complete_all(&g_base) >= 1); + CU_ASSERT(ctx.done && ctx.rc == 0); + free(page); +} + +/* read one page through the proxy (completes the backing read inline) */ +static void +ut_read_page(uint64_t lba, uint8_t *dst) +{ + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + + g_proxy->read(g_proxy, NULL, dst, lba, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == 0); +} + +static uint8_t * +ut_disk_at(uint64_t lba) +{ + return g_buf + lba * UT_BLOCKLEN; +} + +static bool +ut_mem_is_zero(const uint8_t *p, uint64_t len) +{ + uint64_t i; + + for (i = 0; i < len; i++) { + if (p[i] != 0) { + return false; + } + } + return true; +} + +/* ------------------------------------------------------------------ */ +/* U1: append/ack ordering (I1) + entry bytes on disk */ + +static void +test_append_ack_ordering(void) +{ + struct ut_cb_ctx ctx = {}; + uint8_t *page = malloc(BS_MD_JOURNAL_PAGE_SIZE); + uint8_t expect[BS_MD_JOURNAL_PAGE_SIZE]; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + struct md_journal_entry_hdr hdr; + struct ut_io *io; + uint64_t entry_lba; + + SPDK_CU_ASSERT_FATAL(page != NULL); + ut_journal_setup(); + + ut_fill_page(page, 5, 0xA5); + memcpy(expect, page, sizeof(expect)); + + g_proxy->write(g_proxy, NULL, page, 5, 1, &cb_args, &g_ut_io_opts); + + /* the md write was redirected to the ring, not to LBA 5 */ + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + entry_lba = slot_to_lba(g_jr, 0); + CU_ASSERT(io->type == UT_IO_WRITE); + CU_ASSERT(io->lba == entry_lba); + CU_ASSERT(io->lba_count == 2 * g_jr->blocks_per_page); + + /* I1: caller must not be acknowledged before the journal write + * completes */ + CU_ASSERT(!ctx.done); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == 0); + + /* entry bytes on disk: [header block][md page] */ + memcpy(&hdr, ut_disk_at(entry_lba), sizeof(hdr)); + CU_ASSERT(hdr.magic == BS_MD_JOURNAL_HDR_MAGIC); + CU_ASSERT(hdr.target_lba == 5); + CU_ASSERT(hdr.crc == spdk_crc32c_update(expect, BS_MD_JOURNAL_PAGE_SIZE, 0)); + CU_ASSERT(memcmp(ut_disk_at(entry_lba + g_jr->blocks_per_page), expect, + BS_MD_JOURNAL_PAGE_SIZE) == 0); + + /* the home LBA is untouched so far */ + CU_ASSERT(ut_mem_is_zero(ut_disk_at(5), BS_MD_JOURNAL_PAGE_SIZE)); + + /* the journal keeps its own copy: caller buffer may be reused */ + memset(page, 0xFF, BS_MD_JOURNAL_PAGE_SIZE); + { + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + + ut_read_page(5, rbuf); + CU_ASSERT(memcmp(rbuf, expect, BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + CU_ASSERT(g_jr->used_slots == 1); + CU_ASSERT(g_jr->mem_head == 1 && g_jr->disk_head == 1); + + free(page); + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U2: read overlay serves the journaled page while home is stale */ + +static void +test_read_overlay(void) +{ + uint8_t expect[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t stale[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + + ut_journal_setup(); + + /* stale home content */ + ut_fill_page(stale, 7, 0x11); + memcpy(ut_disk_at(7), stale, BS_MD_JOURNAL_PAGE_SIZE); + + ut_fill_page(expect, 7, 0xB7); + ut_append_page(7, 0xB7); + + /* home is still stale (no drain has run) ... */ + CU_ASSERT(memcmp(ut_disk_at(7), stale, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + /* ... but the proxy read returns the journaled page (payload form) */ + memset(rbuf, 0, sizeof(rbuf)); + ut_read_page(7, rbuf); + CU_ASSERT(memcmp(rbuf, expect, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + /* iov form (readv) gets the same overlay */ + { + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + uint8_t part1[1024], part2[3072]; + struct iovec iov[2] = { + { .iov_base = part1, .iov_len = sizeof(part1) }, + { .iov_base = part2, .iov_len = sizeof(part2) }, + }; + + g_proxy->readv(g_proxy, NULL, iov, 2, 7, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == 0); + CU_ASSERT(memcmp(part1, expect, sizeof(part1)) == 0); + CU_ASSERT(memcmp(part2, expect + sizeof(part1), sizeof(part2)) == 0); + } + + /* a page that was never journaled comes from home untouched */ + { + uint8_t other[BS_MD_JOURNAL_PAGE_SIZE]; + + ut_fill_page(other, 8, 0x22); + memcpy(ut_disk_at(8), other, BS_MD_JOURNAL_PAGE_SIZE); + memset(rbuf, 0, sizeof(rbuf)); + ut_read_page(8, rbuf); + CU_ASSERT(memcmp(rbuf, other, BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U3: drain writes home, zeroes both entry blocks, drops the dict */ + +static void +test_drain(void) +{ + uint8_t expect[BS_MD_JOURNAL_PAGE_SIZE]; + uint64_t entry_lba; + struct ut_io *io; + + ut_journal_setup(); + + ut_fill_page(expect, 9, 0xC3); + ut_append_page(9, 0xC3); + CU_ASSERT(g_jr->used_slots == 1); + entry_lba = slot_to_lba(g_jr, 0); + + /* drain step 1: home write from the in-memory copy */ + poll_threads(); + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->type == UT_IO_WRITE); + CU_ASSERT(io->lba == 9); + CU_ASSERT(io->lba_count == g_jr->blocks_per_page); + + /* I2: the entry must not be zeroed before the home write completed */ + CU_ASSERT(!ut_mem_is_zero(ut_disk_at(entry_lba), BS_MD_JOURNAL_ENTRY_BYTES)); + CU_ASSERT(ut_dev_complete_one(&g_base)); + + /* drain step 2: zero both blocks of the entry */ + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->type == UT_IO_WRITE_ZEROES); + CU_ASSERT(io->lba == entry_lba); + CU_ASSERT(io->lba_count == 2 * g_jr->blocks_per_page); + CU_ASSERT(ut_dev_complete_one(&g_base)); + + /* end state: page home, entry zeroed, dict dropped, slot freed */ + CU_ASSERT(memcmp(ut_disk_at(9), expect, BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(ut_mem_is_zero(ut_disk_at(entry_lba), BS_MD_JOURNAL_ENTRY_BYTES)); + spdk_spin_lock(&g_jr->lock); + CU_ASSERT(dict_get(g_jr, 9) == JOURNAL_SLOT_INVALID); + spdk_spin_unlock(&g_jr->lock); + CU_ASSERT(g_jr->used_slots == 0); + CU_ASSERT(g_jr->mem_tail == 1 && g_jr->disk_tail == 1); + + /* a read now comes from home */ + { + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + + ut_read_page(9, rbuf); + CU_ASSERT(memcmp(rbuf, expect, BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U4: two appends to the same LBA: the older entry is drained without + * a home write (supersede-coalescing), the newer one lands home */ + +static void +test_supersede_coalescing(void) +{ + uint8_t expect_b[BS_MD_JOURNAL_PAGE_SIZE]; + struct ut_io *io; + + ut_journal_setup(); + + ut_append_page(11, 0x0A); /* version A -> slot 0 */ + ut_append_page(11, 0x0B); /* version B -> slot 1 */ + ut_fill_page(expect_b, 11, 0x0B); + CU_ASSERT(g_jr->used_slots == 2); + + /* first drain step on slot 0 must be a zero, not a home write */ + poll_threads(); + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->type == UT_IO_WRITE_ZEROES); + CU_ASSERT(io->lba == slot_to_lba(g_jr, 0)); + CU_ASSERT(ut_dev_complete_one(&g_base)); + CU_ASSERT(g_jr->used_slots == 1); + /* the dictionary still serves version B */ + spdk_spin_lock(&g_jr->lock); + CU_ASSERT(dict_get(g_jr, 11) == 1); + spdk_spin_unlock(&g_jr->lock); + + /* second drain: home write of B, then zero of slot 1 */ + poll_threads(); + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->type == UT_IO_WRITE); + CU_ASSERT(io->lba == 11); + ut_settle(&g_base); + + CU_ASSERT(memcmp(ut_disk_at(11), expect_b, BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(g_jr->used_slots == 0); + /* exactly one home write ever hit LBA 11 */ + CU_ASSERT(g_base.writes_completed[UT_IO_WRITE] == 2 /* journal entries */ + 1 /* home */); + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U5: ring full (8191 entries) stalls the next append; a drained slot + * lets it proceed */ + +static void +test_ring_full_stall(void) +{ + struct ut_cb_ctx *ctxs; + struct spdk_bs_dev_cb_args *cbs; + uint8_t *page = malloc(BS_MD_JOURNAL_PAGE_SIZE); + struct ut_cb_ctx stall_ctx = {}; + struct spdk_bs_dev_cb_args stall_cb = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &stall_ctx }; + uint32_t i, fill = BS_MD_JOURNAL_NUM_SLOTS - 1; /* 8191: guard slot keeps one free */ + + SPDK_CU_ASSERT_FATAL(page != NULL); + ctxs = calloc(fill, sizeof(*ctxs)); + cbs = calloc(fill, sizeof(*cbs)); + SPDK_CU_ASSERT_FATAL(ctxs != NULL && cbs != NULL); + + ut_journal_setup(); + + for (i = 0; i < fill; i++) { + cbs[i].cb_fn = ut_io_cb; + cbs[i].cb_arg = &ctxs[i]; + ut_fill_page(page, i, 0x55); + g_proxy->write(g_proxy, NULL, page, i, 1, &cbs[i], &g_ut_io_opts); + /* strict FIFO: exactly one journal write in flight */ + CU_ASSERT(g_base.pending == 1); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctxs[i].done && ctxs[i].rc == 0); + } + CU_ASSERT(g_jr->used_slots == fill); + CU_ASSERT(ring_full(g_jr)); + + /* the 8192nd append must stall: no base IO, no ack */ + ut_fill_page(page, 4000, 0x77); + g_proxy->write(g_proxy, NULL, page, 4000, 1, &stall_cb, &g_ut_io_opts); + CU_ASSERT(g_base.pending == 0); + CU_ASSERT(!stall_ctx.done); + + /* one drain cycle frees one slot -> the stalled append proceeds */ + poll_threads(); /* home write of entry 0 */ + CU_ASSERT(ut_dev_complete_one(&g_base)); + CU_ASSERT(ut_dev_complete_one(&g_base)); /* zero of entry 0 */ + /* freeing the slot pumped the waiting append */ + CU_ASSERT(g_base.pending == 1); + CU_ASSERT(!stall_ctx.done); /* I1 still holds */ + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(stall_ctx.done && stall_ctx.rc == 0); + CU_ASSERT(g_jr->used_slots == fill); + + free(page); + free(ctxs); + free(cbs); + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* recovery helpers (U6-U9) */ + +/* tear down only the journal object, keeping the flat buffer intact + * (simulates the node going away without a clean drain) */ +static void +ut_journal_abandon(void) +{ + CU_ASSERT(g_base.pending == 0); + g_proxy->destroy(g_proxy); + CU_ASSERT(g_base.destroyed); + g_jr = NULL; + g_proxy = NULL; +} + +/* bring up a journal on the existing buffer and run recovery */ +static void +ut_journal_recover(int expected_rc) +{ + struct ut_cb_ctx start_ctx = {}; + + ut_dev_init(&g_base, g_buf); + g_jr = NULL; + g_proxy = bs_md_journal_dev_create(&g_base.bs_dev, &g_jr); + SPDK_CU_ASSERT_FATAL(g_proxy != NULL && g_jr != NULL); + + bs_md_journal_start(g_jr, false, ut_start_cb, &start_ctx); + /* the parallel ring scan is outstanding now */ + CU_ASSERT(g_base.pending == BS_MD_JOURNAL_RECOVERY_QDEPTH); + CU_ASSERT(!start_ctx.done); + ut_dev_complete_all(&g_base); + CU_ASSERT(start_ctx.done); + CU_ASSERT(start_ctx.rc == expected_rc); + + if (expected_rc == 0) { + /* recovery must arm interception for the whole proxy range + * before the caller's first (super block) read; the caller + * then tightens the limit after parsing the super */ + CU_ASSERT(g_jr->md_limit_lba == g_jr->journal_start_lba); + bs_md_journal_enable(g_jr, UT_MD_LIMIT_LBA); + } +} + +/* ------------------------------------------------------------------ */ +/* U6: recovery rebuilds tail/head/dict from the valid run and the + * drain poller works the backlog off */ + +static void +test_recovery_basic(void) +{ + uint8_t expect[3][BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + uint32_t i; + + /* recovery over a completely zeroed ring finds nothing */ + g_buf = calloc(1, UT_DEV_SIZE); + SPDK_CU_ASSERT_FATAL(g_buf != NULL); + ut_journal_recover(0); + CU_ASSERT(g_jr->used_slots == 0); + CU_ASSERT(g_jr->mem_head == g_jr->mem_tail); + ut_journal_abandon(); + free(g_buf); + + /* populate three entries, then "crash" (no drain) */ + ut_journal_setup(); + for (i = 0; i < 3; i++) { + ut_fill_page(expect[i], 10 + i, 0x60 + i); + ut_append_page(10 + i, 0x60 + i); + } + ut_journal_abandon(); + + ut_journal_recover(0); + CU_ASSERT(g_jr->used_slots == 3); + CU_ASSERT(g_jr->mem_tail == 0 && g_jr->disk_tail == 0); + CU_ASSERT(g_jr->mem_head == 3 && g_jr->disk_head == 3); + + /* every acked page is served via the rebuilt dictionary */ + for (i = 0; i < 3; i++) { + CU_ASSERT(ut_mem_is_zero(ut_disk_at(10 + i), BS_MD_JOURNAL_PAGE_SIZE)); + ut_read_page(10 + i, rbuf); + CU_ASSERT(memcmp(rbuf, expect[i], BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + /* the drain poller works the backlog off in the background */ + ut_settle(&g_base); + CU_ASSERT(g_jr->used_slots == 0); + for (i = 0; i < 3; i++) { + CU_ASSERT(memcmp(ut_disk_at(10 + i), expect[i], BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(ut_mem_is_zero(ut_disk_at(slot_to_lba(g_jr, i)), + BS_MD_JOURNAL_ENTRY_BYTES)); + } + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U7: recovery treats a torn (bad-crc) trailing entry and zeroed + * entries as empty */ + +static void +test_recovery_torn_entry(void) +{ + uint8_t expect[2][BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + uint64_t torn_page_lba; + uint32_t i; + + ut_journal_setup(); + for (i = 0; i < 2; i++) { + ut_fill_page(expect[i], 10 + i, 0x70 + i); + ut_append_page(10 + i, 0x70 + i); + } + ut_append_page(12, 0x72); + torn_page_lba = slot_to_lba(g_jr, 2); + ut_journal_abandon(); + + /* tear the last entry: corrupt one byte of its payload page, as a + * power loss mid-write would (checksum mismatch) */ + g_buf[(torn_page_lba + 1) * UT_BLOCKLEN + 100] ^= 0xFF; + + ut_journal_recover(0); + /* the torn entry is empty by definition: run = slots 0..1 */ + CU_ASSERT(g_jr->used_slots == 2); + CU_ASSERT(g_jr->mem_tail == 0); + CU_ASSERT(g_jr->mem_head == 2); + spdk_spin_lock(&g_jr->lock); + CU_ASSERT(dict_get(g_jr, 12) == JOURNAL_SLOT_INVALID); + spdk_spin_unlock(&g_jr->lock); + + for (i = 0; i < 2; i++) { + ut_read_page(10 + i, rbuf); + CU_ASSERT(memcmp(rbuf, expect[i], BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + /* the torn page reads from home (never acked, so any content is + * legal; home was never written -> zeroes) */ + ut_read_page(12, rbuf); + CU_ASSERT(ut_mem_is_zero(rbuf, BS_MD_JOURNAL_PAGE_SIZE)); + + ut_settle(&g_base); + CU_ASSERT(g_jr->used_slots == 0); + CU_ASSERT(memcmp(ut_disk_at(10), expect[0], BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(memcmp(ut_disk_at(11), expect[1], BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(ut_mem_is_zero(ut_disk_at(12), BS_MD_JOURNAL_PAGE_SIZE)); + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U8: recovery duplicates: for the same LBA the later ring position + * wins in the dictionary */ + +static void +test_recovery_duplicates(void) +{ + uint8_t expect_b[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + + ut_journal_setup(); + ut_append_page(10, 0x0A); /* older, slot 0 */ + ut_append_page(10, 0x0B); /* newer, slot 1 */ + ut_fill_page(expect_b, 10, 0x0B); + ut_journal_abandon(); + + ut_journal_recover(0); + CU_ASSERT(g_jr->used_slots == 2); + spdk_spin_lock(&g_jr->lock); + CU_ASSERT(dict_get(g_jr, 10) == 1); /* later ring position wins */ + spdk_spin_unlock(&g_jr->lock); + + ut_read_page(10, rbuf); + CU_ASSERT(memcmp(rbuf, expect_b, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + ut_settle(&g_base); + CU_ASSERT(memcmp(ut_disk_at(10), expect_b, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U9: power-off simulation: snapshot the device mid-workload (with the + * in-flight journal write torn), recover on the snapshot: every acked + * write is recoverable, the unacked one vanishes */ + +static void +test_power_off_simulation(void) +{ + uint8_t acked[10][BS_MD_JOURNAL_PAGE_SIZE]; /* newest acked content per lba 20..29 */ + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t *snap; + uint8_t *page = malloc(BS_MD_JOURNAL_PAGE_SIZE); + struct ut_cb_ctx unacked_ctx = {}; + struct spdk_bs_dev_cb_args unacked_cb = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &unacked_ctx }; + struct ut_io *io; + uint32_t r, i; + + SPDK_CU_ASSERT_FATAL(page != NULL); + ut_journal_setup(); + + /* three rounds of md updates over ten pages, all acked */ + for (r = 0; r < 3; r++) { + for (i = 0; i < 10; i++) { + uint8_t seed = (uint8_t)(0x90 + 16 * r + i); + + ut_append_page(20 + i, seed); + ut_fill_page(acked[i], 20 + i, seed); + } + } + + /* one more write is issued but its journal write never completes: + * the caller was never acknowledged */ + ut_fill_page(page, 25, 0xEE); + g_proxy->write(g_proxy, NULL, page, 25, 1, &unacked_cb, &g_ut_io_opts); + CU_ASSERT(g_base.pending == 1); + CU_ASSERT(!unacked_ctx.done); + + /* power-off: snapshot the device as it is, with the in-flight + * entry torn (only its first 512 bytes made it to the platter) */ + snap = malloc(UT_DEV_SIZE); + SPDK_CU_ASSERT_FATAL(snap != NULL); + memcpy(snap, g_buf, UT_DEV_SIZE); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL && io->type == UT_IO_WRITE); + memcpy(snap + io->lba * UT_BLOCKLEN, io->wdata, 512); + + /* let the old world finish cleanly and switch to the snapshot */ + ut_settle(&g_base); + CU_ASSERT(unacked_ctx.done); + g_proxy->destroy(g_proxy); + free(g_buf); + g_buf = snap; + + ut_journal_recover(0); + /* 30 acked entries; the torn 31st is empty by definition */ + CU_ASSERT(g_jr->used_slots == 30); + CU_ASSERT(g_jr->mem_tail == 0 && g_jr->mem_head == 30); + + /* every acked write is recoverable (newest version per lba) */ + for (i = 0; i < 10; i++) { + ut_read_page(20 + i, rbuf); + CU_ASSERT(memcmp(rbuf, acked[i], BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + /* ... and lands home when the backlog drains */ + ut_settle(&g_base); + CU_ASSERT(g_jr->used_slots == 0); + for (i = 0; i < 10; i++) { + CU_ASSERT(memcmp(ut_disk_at(20 + i), acked[i], BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + free(page); + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U10: proxy geometry: blockcnt shrunk by 64 MB; data-range IO passes + * through untouched */ + +static void +test_proxy_geometry_passthrough(void) +{ + uint8_t data[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + uint64_t data_lba = UT_MD_LIMIT_LBA + 100; + struct ut_io *io; + + ut_journal_setup(); + + /* geometry: the ring is carved out of the top of the device */ + CU_ASSERT(g_proxy->blocklen == UT_BLOCKLEN); + CU_ASSERT(g_proxy->blockcnt == UT_BLOCKCNT - UT_JOURNAL_BLOCKS); + CU_ASSERT(bs_md_journal_ring_lba(g_jr) == UT_BLOCKCNT - UT_JOURNAL_BLOCKS); + CU_ASSERT(bs_md_journal_ring_lba_count(g_jr) == UT_JOURNAL_BLOCKS); + + /* a write above the md limit goes straight to its home LBA */ + ut_fill_page(data, data_lba, 0x44); + g_proxy->write(g_proxy, NULL, data, data_lba, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->type == UT_IO_WRITE); + CU_ASSERT(io->lba == data_lba); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == 0); + CU_ASSERT(memcmp(ut_disk_at(data_lba), data, BS_MD_JOURNAL_PAGE_SIZE) == 0); + CU_ASSERT(g_jr->used_slots == 0); /* nothing journaled */ + + /* a range straddling the md limit is not journaled either */ + { + struct ut_cb_ctx ctx2 = {}; + struct spdk_bs_dev_cb_args cb2 = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx2 }; + uint8_t two_pages[2 * BS_MD_JOURNAL_PAGE_SIZE]; + + memset(two_pages, 0x5A, sizeof(two_pages)); + g_proxy->write(g_proxy, NULL, two_pages, UT_MD_LIMIT_LBA - 1, 2, &cb2, &g_ut_io_opts); + CU_ASSERT(g_base.pending == 1); + io = TAILQ_FIRST(&g_base.io_queue); + SPDK_CU_ASSERT_FATAL(io != NULL); + CU_ASSERT(io->lba == UT_MD_LIMIT_LBA - 1 && io->lba_count == 2); + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx2.done && ctx2.rc == 0); + CU_ASSERT(g_jr->used_slots == 0); + } + + /* data-range reads pass through without overlay interference */ + memset(rbuf, 0, sizeof(rbuf)); + ut_read_page(data_lba, rbuf); + CU_ASSERT(memcmp(rbuf, data, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U11: multi-page (mask-style) append: N entries in FIFO order, one + * single ack once all of them are durable */ + +static void +test_multi_page_append(void) +{ + uint8_t content[3 * BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + struct iovec iov[2]; + struct md_journal_entry_hdr hdr; + uint32_t i; + + ut_journal_setup(); + + for (i = 0; i < sizeof(content); i++) { + content[i] = (uint8_t)(i * 7 + 3); + } + /* 12K in two unaligned iovs: 8K + 4K */ + iov[0].iov_base = content; + iov[0].iov_len = 2 * BS_MD_JOURNAL_PAGE_SIZE; + iov[1].iov_base = content + 2 * BS_MD_JOURNAL_PAGE_SIZE; + iov[1].iov_len = BS_MD_JOURNAL_PAGE_SIZE; + + g_proxy->writev(g_proxy, NULL, iov, 2, 40, 3, &cb_args, &g_ut_io_opts); + + /* page-by-page FIFO: three journal writes, strictly one at a time; + * the caller is acknowledged only after the last one (I1/I3) */ + for (i = 0; i < 3; i++) { + CU_ASSERT(!ctx.done); + CU_ASSERT(g_base.pending == 1); + CU_ASSERT(ut_dev_complete_one(&g_base)); + } + CU_ASSERT(ctx.done && ctx.rc == 0); + CU_ASSERT(g_base.pending == 0); + CU_ASSERT(g_jr->used_slots == 3); + + /* ring order matches issue order (I3) */ + for (i = 0; i < 3; i++) { + memcpy(&hdr, ut_disk_at(slot_to_lba(g_jr, i)), sizeof(hdr)); + CU_ASSERT(hdr.magic == BS_MD_JOURNAL_HDR_MAGIC); + CU_ASSERT(hdr.target_lba == 40 + i); + CU_ASSERT(memcmp(ut_disk_at(slot_to_lba(g_jr, i) + g_jr->blocks_per_page), + content + (uint64_t)i * BS_MD_JOURNAL_PAGE_SIZE, + BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + /* each page is readable through the overlay */ + for (i = 0; i < 3; i++) { + ut_read_page(40 + i, rbuf); + CU_ASSERT(memcmp(rbuf, content + (uint64_t)i * BS_MD_JOURNAL_PAGE_SIZE, + BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + /* and drains home in order */ + ut_settle(&g_base); + for (i = 0; i < 3; i++) { + CU_ASSERT(memcmp(ut_disk_at(40 + i), + content + (uint64_t)i * BS_MD_JOURNAL_PAGE_SIZE, + BS_MD_JOURNAL_PAGE_SIZE) == 0); + } + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ +/* U12: destroy with an in-flight journal write defers the teardown to + * the IO completion (no use-after-free), caller gets -ESHUTDOWN */ + +static void +test_destroy_inflight(void) +{ + uint8_t *page = malloc(BS_MD_JOURNAL_PAGE_SIZE); + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + + SPDK_CU_ASSERT_FATAL(page != NULL); + ut_journal_setup(); + + ut_fill_page(page, 50, 0xD0); + g_proxy->write(g_proxy, NULL, page, 50, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(g_base.pending == 1); + CU_ASSERT(!ctx.done); + + /* destroy with the journal write still in flight: the teardown must + * be deferred until that IO completes */ + g_proxy->destroy(g_proxy); + CU_ASSERT(!g_base.destroyed); + + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == -ESHUTDOWN); + CU_ASSERT(g_base.destroyed); + + free(page); + free(g_buf); + g_buf = NULL; + g_jr = NULL; + g_proxy = NULL; +} + +/* ------------------------------------------------------------------ */ +/* U13: read vs drain race: a home read issued while the page is still + * journaled must return the journaled content even when the drain + * completes (home write + zero + dict drop) before the read does and + * the device serves the read from the pre-drain page content */ + +static void +test_read_vs_drain_race(void) +{ + uint8_t stale[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t expect[BS_MD_JOURNAL_PAGE_SIZE]; + uint8_t rbuf[BS_MD_JOURNAL_PAGE_SIZE]; + struct ut_cb_ctx ctx = {}; + struct spdk_bs_dev_cb_args cb_args = { .cb_fn = ut_io_cb, .channel = NULL, .cb_arg = &ctx }; + + ut_journal_setup(); + + ut_fill_page(stale, 33, 0x11); + memcpy(ut_disk_at(33), stale, BS_MD_JOURNAL_PAGE_SIZE); + ut_fill_page(expect, 33, 0x99); + ut_append_page(33, 0x99); /* journaled, acked, not drained */ + + /* the read is issued now — the mock snapshots the still-stale home */ + g_proxy->read(g_proxy, NULL, rbuf, 33, 1, &cb_args, &g_ut_io_opts); + CU_ASSERT(!ctx.done); + + /* the drain overtakes the in-flight read: home write, then entry + * zeroing, dict entry dropped, slot freed */ + poll_threads(); + CU_ASSERT(ut_dev_complete_first_of(&g_base, UT_IO_WRITE)); + CU_ASSERT(ut_dev_complete_first_of(&g_base, UT_IO_WRITE_ZEROES)); + spdk_spin_lock(&g_jr->lock); + CU_ASSERT(dict_get(g_jr, 33) == JOURNAL_SLOT_INVALID); + spdk_spin_unlock(&g_jr->lock); + CU_ASSERT(g_jr->used_slots == 0); + + /* now the read completes with its pre-drain (stale) device data: + * the issue-time snapshot must win */ + CU_ASSERT(ut_dev_complete_all(&g_base) == 1); + CU_ASSERT(ctx.done && ctx.rc == 0); + CU_ASSERT(memcmp(rbuf, expect, BS_MD_JOURNAL_PAGE_SIZE) == 0); + + ut_journal_teardown(); +} + +/* ------------------------------------------------------------------ */ + +int +main(int argc, char **argv) +{ + CU_pSuite suite = NULL; + unsigned int num_failures; + + CU_initialize_registry(); + + suite = CU_add_suite("blob_md_journal", NULL, NULL); + + CU_ADD_TEST(suite, test_append_ack_ordering); + CU_ADD_TEST(suite, test_read_overlay); + CU_ADD_TEST(suite, test_drain); + CU_ADD_TEST(suite, test_supersede_coalescing); + CU_ADD_TEST(suite, test_ring_full_stall); + CU_ADD_TEST(suite, test_recovery_basic); + CU_ADD_TEST(suite, test_recovery_torn_entry); + CU_ADD_TEST(suite, test_recovery_duplicates); + CU_ADD_TEST(suite, test_power_off_simulation); + CU_ADD_TEST(suite, test_proxy_geometry_passthrough); + CU_ADD_TEST(suite, test_multi_page_append); + CU_ADD_TEST(suite, test_destroy_inflight); + CU_ADD_TEST(suite, test_read_vs_drain_race); + + allocate_threads(1); + set_thread(0); + + num_failures = spdk_ut_run_tests(argc, argv, NULL); + + free_threads(); + + return num_failures; +} diff --git a/test/unit/unittest.sh b/test/unit/unittest.sh index 88a4ae039c9..6d0769c03c5 100755 --- a/test/unit/unittest.sh +++ b/test/unit/unittest.sh @@ -40,6 +40,7 @@ function unittest_blob() { $valgrind $testdir/lib/blob/blob.c/blob_ut fi $valgrind $testdir/lib/blob/blob_bdev.c/blob_bdev_ut + $valgrind $testdir/lib/blob/blob_md_journal.c/blob_md_journal_ut $valgrind $testdir/lib/blobfs/tree.c/tree_ut $valgrind $testdir/lib/blobfs/blobfs_async_ut/blobfs_async_ut # blobfs_sync_ut hangs when run under valgrind, so don't use $valgrind