OPFS support for LadybugDB Wasm ?

[kbatchu]

Hello,
Are there any concrete plans for implementing OPFS support for LadybugDB Wasm (like DuckDB)?
Integrating OPFS support into LadybugDB would be a game-changer for WebAI, especially when paired with DuckDB for a complete local-first data stack.
Thanks
Kiran

[adsharma]

Some additional context about different storage options for wasm:

https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html

[adsharma]

LadybugDB/ladybug-wasm#4 has been merged. It should be in tonight's nightly build. Please try it out and let us know how it works for you.

[kbatchu]

Hi Arun, thanks for merging the OPFS PR (#4 in ladybug-wasm) so quickly!

I'm trying to use the new mountOpfs() API in my project, but the current npm release (@lbug/lbug-wasm@0.13.1) doesn't include it yet. I checked npm view @lbug/lbug-wasm dist-tags and only see { latest: '0.13.1' } — no nightly tag.

Could you let me know:

What's the exact npm install command to get the nightly build with OPFS support?
Is there a nightly or next dist-tag I should be using, or will it be published as a new version number (e.g. 0.14.0)?
Thanks!

[adsharma]

The package name has changed. @ladybugdb/core and @ladybugdb/wasm. You should see 0.15.x

[adsharma]

Re: how to test the nightly builds

You'd locate the most recent nightly run such as:

https://github.com/LadybugDB/ladybug/actions/runs/23487507898

It should have a deploy-nodejs and deploy-wasm step and that step has the artifacts you can download and test locally.

Need to look into the build/test failure from yesterday. There was no deployment. Rerunning the failed job.

[kbatchu]

Thanks for the new @ladybugdb/wasm-core package! I installed the latest nightly (0.15.3-dev.1) and confirmed the mountOpfs() API is available on the FS class.

However, calling fs.mountOpfs('/persist') returns:
"OPFS is not available. Ensure the build was compiled with -sWASMFS and -pthread flags."
This happens in both the default and multithreaded variants. It looks like the API surface is merged, but neither variant is actually compiled with -sWASMFS — so FS.filesystems.OPFS is undefined at runtime.
Could a WASMFS-enabled variant be added to the nightly build? Something like a wasmfs/ directory alongside the existing multithreaded/ variant?
For now I'm using IDBFS with syncfs() which works well, but OPFS would eliminate the manual sync overhead.

Thanks

[adsharma]

#321

[kbatchu]

Hi Arun, thanks for the quick turnaround on PR #321!

I noticed the deploy-wasm step in the latest nightly run (#383) completed in 0s — it looks like it may have been skipped or the WASM artifacts weren't deployed. Could you confirm whether the nightly build from tonight will include the WASMFS fix from #321, or would it help if you manually trigger a build?

Thanks!

[adsharma]

Enabling wasmfs broke tests. #322 disables wasmfs for tests (which means that it gets no test coverage, which is suboptimal).

Manual build here:
https://github.com/LadybugDB/ladybug/actions/runs/23557931989

[kbatchu]

Hi Arun, I downloaded the WASM artifact from manual build #385 and tested it. mountOpfs() still returns "OPFS is not available. Ensure the build was compiled with -sWASMFS and -pthread flags." — FS.filesystems.OPFS is undefined at runtime. It seems this build's .wasm binary wasn't compiled with -sWASMFS. Could you verify the build flags in the CI output? In the meantime, IDBFS fallback is working fine.
Thanks

[adsharma]

I just downloaded that build and had claude check it:

SHA256 digest of uploaded artifact zip is ca32f1cf164fcbf9c08fba0b18d69ce9a556814e93aa0b65c7392a305e6bfc7b
Finalizing artifact upload
Artifact lbug-deploy-wasm.zip successfully finalized. Artifact ID 6109252548
Artifact lbug-deploy-wasm has been successfully uploaded! Final size is 15598668 bytes. Artifact ID is 6109252548
Artifact download URL: https://github.com/LadybugDB/ladybug/actions/runs/23557931989/artifacts/6109252548

The evidence is very clear. The wasm binary contains extensive WASMFS symbols and source paths from the emsdk
wasmfs library.

Yes, this package was compiled with WASMFS. The evidence is definitive:

From the .wasm binary (nodejs/lbug/lbug_wasm.wasm)

Imported/exported WASMFS symbols found:

• _wasmfs_stdin_get_char, _wasmfs_get_preloaded_file_size, _wasmfs_copy_preloaded_file_data
• _wasmfs_node_open/close/read/write/ftruncate/stat/fstat, etc.
• _wasmfs_write_file, _wasmfs_get_cwd, _wasmfs_mknod
• wasmfs_create_memory_backend, twasmfs_flush

Debug source paths embedded in the binary:

  /emsdk/emscripten/system/lib/wasmfs/syscalls.cpp
  /emsdk/emscripten/system/lib/wasmfs/wasmfs.cpp
  /emsdk/emscripten/system/lib/wasmfs/special_files.cpp
  /emsdk/emscripten/system/lib/wasmfs/file.cpp
  /emsdk/emscripten/system/lib/wasmfs/file_table.cpp
  /emsdk/emscripten/system/lib/wasmfs/backends/node_backend.cpp

C++ mangled class names for wasmfs::MemoryBackend, wasmfs::NodeBackend, wasmfs::SpecialFiles, etc. are all
present.

Also confirmed in JS glue files

The JS files (index.js, lbug_wasm_worker.js, etc.) also reference WASMFS symbols directly.

Summary

The package was compiled with -sWASMFS (Emscripten's WasmFS flag), which links in libwasmfs (equivalent to
-lwasmfs). The WasmFS NodeBackend is also compiled in, meaning it supports mapping WasmFS paths to the host
Node.js filesystem.

[kbatchu]

I am using Claude as well. Claude confirmed the binary does contain WASMFS (wasmfs_create_memory_backend, NodeBackend), but the OPFS backend is missing — there are no opfs_backend or wasmfs_create_opfs_backend symbols in the .wasm binary. This explains why FS.filesystems.OPFS is undefined at runtime. It looks like the build uses -sWASMFS but not -sWASMFS_OPFS — could you check the CMakeLists.txt?

Thanks for your patience.

[adsharma]

That's right - there is no -sWASMFS_OPFS. Will add.

[adsharma]

Pushed. Should show up in the next build in a few hours

[adsharma]

#330

There is no such thing as -sWASMFS_OPFS

[kbatchu]

Thanks for the quick turnaround on #330! To clarify — you're right that -sWASMFS_OPFS isn't a valid Emscripten settings flag. The actual mechanism is linking -lopfs (the OPFS backend library), which is what #330 adds. Once this merges and lands in a nightly, I'll test the full persistence round-trip (mountOpfs → CSV import → page refresh → restore) and report back.
Thanks

[kbatchu]

OPFS mount fails with WASMFS build — JS wrapper uses legacy FS API

I built LadybugDB WASM from main (post PR #330 merge) and tested OPFS mounting. Here's what I found:

Single-threaded build: The -lopfs.js flag is linked, but wasmfsOPFSCreateAsyncAccessHandle and other OPFS symbols end up in missingLibrarySymbols. OPFS requires pthreads to function.

Multi-threaded build (with -pthread): The OPFS backend is compiled in ("OPFS" appears in unexportedSymbols), but at runtime FSMountOpfs() fails with:

Cannot read properties of undefined (reading 'OPFS')
Root cause: FSMountOpfs in lbug_wasm_worker.js (line ~7915) accesses FS.filesystems.OPFS, which is a legacy Emscripten FS API. But the build uses -sWASMFS, and WASMFS does not populate FS.filesystems. The OPFS backend is linked at the C/WASM level but the JS wrapper doesn't know how to access it through WASMFS's backend system.

The multi-threaded build also exhausts the thread pool (-sPTHREAD_POOL_SIZE=8) when loaded through the single-worker architecture.

Environment: Emscripten 5.0.4, Windows, Chrome, built from main @ commit post-PR #330.

Would appreciate guidance on the right WASMFS API to mount OPFS, or if this needs a fix in the ladybug-wasm JS wrapper code.
Thanks

[adsharma]

Let's ignore the single threaded build and focus on the multi-threaded build, since wasmfs requires pthreads to function.

I don't know the answers to these questions. Will research and get back.

The feedback loop could be faster/tighter if you're able to clone the git repo with the wasm submodule and replicate the nightly builds on your end.

[adsharma]

@kbatchu created a couple of PRs and kicked off the build

https://github.com/LadybugDB/ladybug/actions/runs/23826241794

If this doesn't we'll have to revisit this at a later point, since there are other pressing priorities.

[adsharma]

Further discussion here: emscripten-core/emscripten#25444

[adsharma]

Test is passing:

    OPFS persistent storage
      OPFS persistent storage
        - mountOpfs succeeds in multi-threaded mode
        - can create a database at an OPFS path
        - persists data written to an OPFS-backed database within a session

[kbatchu]

Thanks for the quick turnaround on the OPFS fix in v0.15.4-dev.1! The JS wrapper rewrite is correct — FSMountOpfs now uses the WasmFS C API. The mount succeeds, but database file creation crashes:

Unexpected child type
UNREACHABLE executed
at /emsdk/emscripten/system/lib/wasmfs/backends/opfs_backend.cpp:292
Root cause: wasmfsOPFSProxyFinish is compiled as a no-op in the browser worker:

// In lbug_wasm_worker.js (v0.15.4-dev.1)
var wasmfsOPFSProxyFinish = (ctx) => {
};
It should be calling _emscripten_proxy_finish(ctx) — that's the signal that tells the C++ pthread the async JS OPFS operation is done. Without it, the C++ thread reads childType before JS writes it (gets 0 instead of 1 or 2), hitting WASMFS_UNREACHABLE.

In Emscripten's source (src/lib/libwasmfs_opfs.js), this function is guarded by #if PTHREADS:

$wasmfsOPFSProxyFinish: (ctx) => {
#if PTHREADS
_emscripten_proxy_finish(ctx);
#endif
},
The build uses pthreads (SharedArrayBuffer + Web Workers are present), but the OPFS JS glue was apparently compiled without the PTHREADS preprocessor flag, so the _emscripten_proxy_finish call was stripped.

Fix: Ensure -sPTHREADS (or -pthread) is set when the Emscripten linker processes -lopfs.js. This might mean the flag needs to be in the link step that pulls in libwasmfs_opfs.js, not just the compile step.

Reproduction:

await lbug.init();
const fs = new lbug.FS();
await fs.mountOpfs('/persist'); // ✅ succeeds
const database = new lbug.Database('/persist/mydb');
const conn = new lbug.Connection(database);
await conn.init(); // 💥 crashes
Environment: Chrome 135, Windows 11, crossOriginIsolated: true, @ladybugdb/wasm-core@0.15.4-dev.1

[adsharma]

@kbatchu any chance you can send this as a PR against the ladybug-wasn repo?

[kbatchu]

#360

[adsharma]

@kbatchu everything working ok here? Thinking about 0.15.4 next week.

[kbatchu]

Thanks for checking in — quick status:

PR #360 (thanks again for landing it) resolved the original OPFS crash on the multithreaded build. However, I'm still on v0.15.3-dev.1 with IDBFS because two follow-on blockers kept me from adopting the multithreaded OPFS path:

1. IDBFS dropped from the default/WasmFS build — FS.mountIdbfs('/data') throws "Cannot read properties of undefined (reading 'IDBFS')". This removed my fallback path for single-threaded contexts.
2. pthread pool exhaustion in the multithreaded build — in my tab I run LadybugDB alongside DuckDB-WASM and a WebLLM runtime. Combined worker count (~24 with hardwareConcurrency + PTHREAD_POOL_SIZE) exceeds the browser's ~20-per-origin limit, and OPFS mount hangs on "thread pool is exhausted." I tested pthreadPoolSize 8, 16, and 0 — all failed. Tracked this as Multithreaded WASM build: -sPTHREAD_POOL_SIZE too low for multi-engine apps #363.

I saw @dukedorje opened #399 on April 20 describing exactly these two issues, with concrete one-flag fixes (-lidbfs.js for IDBFS, -sPTHREAD_POOL_SIZE_STRICT=0 for on-demand thread spawning). If 0.15.4 picks those up, my blockers go away.

Happy to validate a 0.15.4 pre-release against the specific three-engine configuration I'm running (LadybugDB + DuckDB-WASM + WebLLM in one tab) — that's a stress case neither #363 nor #399 explicitly exercises, and it's where the pthread contention actually bites. Just point me at a tag or nightly when it's ready.

[adsharma]

https://ladybugdb.github.io/wasm-shell/ is using opfs now.