Reopening the same database file in the same process fails WAL replay when the WAL contains records >4KB (graceful close, not crash)

[qtchen-vault]

Summary

Opening the same .lbug file with a second Database instance in the same process (after gracefully closing the first) fails during WAL replay, but only when the WAL contains a single record larger than ~4KB (e.g. a STRING >4000 bytes or a FLOAT[1024] embedding). The database file itself is not corrupted — reopening the same file from an independent process succeeds.

The failure only reproduces when the first Databases C++ object is still alive (i.e. not yet garbage-collected). Forcing a GC between the two opens makes the second open succeed.

Environment

• @ladybugdb/core-linux-x64 v0.19.0 (Node binding), Node 24.17
• Linux x86_64, 11 GB RAM, ulimit -v unlimited
• Node binding defaults: throwOnWalReplayFailure = true, enableChecksums = true

Minimal reproduction

const lbug = require("@ladybugdb/core-linux-x64");
const fs = require("fs");
const P = "/tmp/repro/M.lbug";   // fresh dir

// Step 1: write a row whose WAL record exceeds ~4KB, then close gracefully
{
  const d = new lbug.Database(P);
  const c = new lbug.Connection(d); c.initSync();
  c.querySync("CREATE NODE TABLE T (id INT64 PRIMARY KEY, content STRING)");
  c.querySync(`CREATE (:T {id:1, content:"${"x".repeat(5000)}"})`); // record > 4KB
  c.close(); d.close();   // graceful close, no CHECKPOINT
}

// Step 2: reopen the SAME file in the SAME process (read-only probe)
try {
  const d2 = new lbug.Database(P, 0, true /*enableCompression*/, true /*readOnly*/);
  const c2 = new lbug.Connection(d2); c2.initSync();
  console.log("OK");
} catch (e) {
  console.log("FAILED:", e.message);
}

Observed error (varies with record size / timing):

Error: Storage exception: Checksum verification failed, the WAL file is corrupted.

or

Error: Runtime exception: Corrupted wal file. Read out invalid WAL record type.

or (rarely, when the 8TB VMRegion mmap of the second instance collides)

Error: Buffer manager exception: Mmap for size 8796093022208 failed.

Findings / evidence

Scenario	Result
open1 writes big record → close → same process reopen	❌ fails
open1 writes big record → close → independent process reopen	✅ OK
open1 close → force global.gc() → reopen same process	✅ OK
open1 on file A → open2 on different file B (same process)	✅ OK
WAL record ≤ ~4000 B (single STRING) → same-process reopen	✅ OK
WAL record > ~4000 B → same-process reopen	❌ fails
Many small records (100 × 200 B, total WAL 33 KB)	✅ OK
File checksum (sha256) before/after the failed reopen	unchanged (file is intact)

Key observations:

1. The threshold is per-record, not total WAL size (100 small records totaling 33 KB reopen fine; a single 5 KB record fails). It aligns with the ChecksumWriter/ChecksumReader entryBuffer initial size (LBUG_PAGE_SIZE, 4096 B) — the resizeBufferIfNeeded path is what exposes the collision.
2. /proc/self/maps shows the first Databases 8 TB VMRegion (MAP_NORESERVE, ~8 TB anonymous mapping) is still present after close(). The Node bindings close() releases the JS reference but the C++ Database object (and its BufferManager/VMRegion) is only destroyed when GC collects it. A second Database instance created before that collides with the leftover buffer-manager state during WAL replay.
3. The DB file is never modified by the failed reopen (verified via sha256), so this is a same-process memory-state bug, not on-disk corruption.
4. Explicit CHECKPOINT before close() makes the same-process reopen succeed — because the WAL is emptied, so there is nothing to replay. (This is why the bug is easy to misattribute to "you must CHECKPOINT after writing FLOAT[] data": the real trigger is same-process multi-instance + a large WAL record.)

Workarounds (confirmed)

• One process ⇔ one Database connection for a given .lbug file (do not open the same file twice in one process), and/or
• Explicit CHECKPOINT after writes (drains the WAL, avoiding the replay path).

Suggested areas to investigate

• Node binding: whether close() should destroy the native Database object eagerly instead of relying on GC finalizers (the C++ Database::~Database() already runs checkpoint when forceCheckpointOnClose is set).
• ChecksumReader/ChecksumWriter resizeBufferIfNeeded path vs. two live BufferManager instances sharing process address space.
• Whether Mmap for size 8796093022208 failed (8 TB = DEFAULT_VM_REGION_MAX_SIZE) on the second instance is a separate address-space exhaustion symptom of the same root cause.

Related but distinct: #714/#716 cover WAL with records >~2 KB after a hard process termination. This report is about a graceful close followed by a same-process reopen — the crash-recovery fix does not cover it.