Cross-process Nostr token transfer is broken — CLI sender→exit→receiver→start loses events; no e2e coverage

[vrogojin]

Summary

While running the manual walkthrough from #218 / PR #222 on real testnet, the basic alice→bob token transfer via the CLI failed end-to-end: bob's sphere payments receive --finalize returns "No new transfers found" indefinitely, even though alice's sphere payments send succeeds and the kind-31113 TOKEN_TRANSFER event reaches the testnet relay.

The same flow works fine in tests/e2e/uxf-send-receive.test.ts because that test runs sender + receiver in the same process — both Sphere instances stay alive, bob's live subscription catches alice's published event in real-time. The CLI is fundamentally different (sender exits, then receiver starts seconds later) and that path has zero e2e coverage.

Repro

Branch: integration/all-fixes (commit 13dc1dc at time of repro)
CLI: @unicity-sphere/cli linked to local sphere-sdk via the standard npm link path.
Relay: wss://nostr-relay.testnet.unicity.network
Walkthrough doc: manual-test-full-recovery.md (PR #222, branch docs/issue-218-full-recovery-manual-test)
Repro script: manual-test-full-recovery.sh on the same branch.

Minimal repro:

# In two separate working dirs, each fresh
cd ~/test-alice
sphere wallet create alice && sphere wallet use alice
SPHERE_ALLOW_MNEMONIC_NON_TTY=1 sphere init --network testnet --nametag alice-XXX
# Save mnemonic
sphere faucet
sphere payments sync
sphere payments send @bob-XXX 20 UCT      # Returns "Status: submitted"

cd ~/test-bob
sphere wallet create bob && sphere wallet use bob
SPHERE_ALLOW_MNEMONIC_NON_TTY=1 sphere init --network testnet --nametag bob-XXX
sphere payments receive --finalize        # "No new transfers found" — should see 20 UCT
sphere balance                            # "No tokens found"

Evidence the bug is on the receiver, not the relay or sender

1. Alice's history.json has the SENT entry with bob's chain pubkey, amount, etc. The L3 burn was accepted by the aggregator.
2. Direct WebSocket query to the relay (anonymous, mimicking the SDK's filter) returns alice's event:

   REQ {kinds:[4,31113,31115,31116], "#p":[bob_transport_pubkey], since:1779443013}
   → EVENT {id:422e59544b1f..., kind:31113, created_at:1779443013, pubkey:927524810c90a65c... (alice)}
   EOSE
   

3. The SDK does send the exact same REQ — confirmed by monkey-patching WebSocket.send and logging every outgoing message:

   ["REQ","sub_3",{"kinds":[4,31113,31115,31116],"#p":["93cd74...bob..."], "since": 1779443013}]
   

4. The SDK does receive the EVENT — confirmed by monkey-patching the WS message handler:

   [WS-RECV] ["EVENT","sub_3", {<alice's event>}]
   [WS-RECV] ["EOSE","sub_3"]
   

5. But [Mux] handleEvent never logs between [Mux] updateSubscriptions: walletSub=sub_3 and [Mux] Wallet subscription EOSE. The event arrives at NostrClient but never reaches MUX dispatch — no decrypt success log, no decrypt failure log, nothing. Even with full SDK debug logging enabled (logger.configure({debug:true})).

Suspicious code

transport/MultiAddressTransportMux.ts:1937 —

async fetchPendingEvents(): Promise<void> {
  // Fetching is handled by subscription — no-op for mux-based adapters
  // The mux subscription already includes this address's pubkey
}

This is the per-address AddressTransportAdapter's fetchPendingEvents. It's a no-op. PaymentsModule.receive() calls this.deps!.transport.fetchPendingEvents() and then snapshots/loads tokens — but the no-op means the snapshot/load doesn't actually wait for any one-shot relay query. The live subscription's onEvent handler runs async, and there's no guarantee it completes (or even fires) before receive() returns.

This isn't the whole story though — patching the no-op to delegate to this.mux.fetchPendingEvents() (which does have a proper bounded since: now-24h-2days one-shot fetch + await for EOSE) did not fix the issue in a quick test. The event still doesn't reach MUX dispatch.

Possible additional contributors (need verification):

• The MUX's persistent subscription uses since=stored_ts. Bob's stored ts is 1779443013 = alice's event's created_at exactly. The relay returns the event (inclusive since), but maybe the SDK's dedup or timing drops it.
• NostrClient.handleEventMessage has a silent try/catch around subscription.listener.onEvent(event) — if the listener throws synchronously, the event is dropped without any log.

Why our e2e tests don't catch this

tests/e2e/uxf-send-receive.test.ts builds both Sphere instances in the same Node process and keeps them alive throughout the test:

const a = await initWallet('alice', aliceTag);
const b = await initWallet('bob', bobTag);
// ...
await a.sphere.payments.send({ recipient: `@${bobTag}`, ... });
await waitFor(async () => {
  try { await b.sphere.payments.receive({ finalize: true }); } catch {}
  return getBalance(b.sphere, SYMBOL).total >= 1000n ? bal : null;
});

Bob's MUX is live and subscribed when alice publishes. The persistent subscription's onEvent fires in real-time. No backfill needed.

The CLI is fundamentally different:

• Alice's process publishes → exits.
• Bob's process starts seconds later, must backfill events from when it was offline.
• This is the cross-process boundary uxf-send-receive cannot exercise.

What we need

A new e2e test that exercises the cross-process pattern. Two options:

Option A — In-process lifecycle (simpler):

1. Init alice's Sphere, send transfer.
2. await alice.destroy().
3. Wait 1–2 seconds.
4. Init bob's Sphere from his mnemonic, fresh storage OR existing-storage carry-over.
5. await bob.payments.receive({finalize:true}).
6. Assert token arrived.

Option B — Subprocess (closer to CLI):

1. Spawn sphere payments send as child process; wait for exit.
2. Spawn sphere payments receive --finalize as child process; capture output.
3. Assert "No new transfers found" is NOT in output and sphere balance shows the expected amount.

Either way, without this test we cannot claim Nostr delivery works on the CLI surface. Same-process assertions are not equivalent.

Refs

• Manual walkthrough doc + repro script: PR docs(manual-test)(#218): full two-peer + daemon + IPFS-only recovery walkthrough #222 (branch docs/issue-218-full-recovery-manual-test)
• Original walkthrough request: docs(manual-test): extend manual-test-drain-fix.md to cover full two-peer + daemon + bidirectional invoice + IPFS-only recovery scenario #218
• Investigation memory snapshot: see internal project_cross_process_nostr_gap.md

Acceptance

• Failing cross-process e2e test added (Option A or B above)
• Root cause isolated (NostrClient listener swallow? subscription map mistiming? handler registration race?)
• Fix lands; same e2e test passes
• Manual walkthrough manual-test-full-recovery.md §§§C/§D run cleanly end-to-end on testnet without surprises

[vrogojin]

Investigation update — Mux race fixed; UXF cross-process delivery is the real bug

⚠️ CRITICAL: cross-process UXF delivery is broken on real testnet. If left unfixed, this defeats the entire UXF upgrade — any sender→receiver where the receiver isn't continuously online loses the transfer.

TL;DR

Initial hypothesis (Mux adapter race) was correct but insufficient. Real-testnet e2e proves a second, deeper bug in the UXF receive path on a freshly-reloaded process. Faucet (TXF-shaped tokens) → wallet works fine; alice→bob (senderUxf=true default → uxf-cid bundle) → wallet does not work after a process restart.

Branch with partial fix + failing-but-correctly-reproducing test: fix/issue-223-cross-process-nostr-delivery (branched from integration/all-fixes @ 13dc1dc).

What's been done (this branch)

1. Root cause #1 — Mux adapter race (FIXED)

Two specialized sub-agents (network-engineer + concurrent-programming) independently converged on the same race:

• Sphere.initializeModules() calls ensureTransportMux → mux.addAddress → updateSubscriptions at line core/Sphere.ts:5021, which subscribes on the live relay socket.
• About 50 lines later (core/Sphere.ts:5061), _payments.initialize() registers the per-address onTokenTransfer handler on the AddressTransportAdapter.
• The relay's stored event for bob lands in the await-gap between those two calls.
• AddressTransportAdapter.dispatchTokenTransfer at transport/MultiAddressTransportMux.ts:1960 iterated an empty transferHandlers Set and silently dropped — asymmetric with dispatchMessage which already had a pendingMessages queue.
• Worse: MultiAddressTransportMux.handleEvent at line 926 adds the event id to processedEventIds before dispatch, poisoning later re-fetch.

Fix:

• Added pendingTransfers / pendingPaymentRequests / pendingPaymentRequestResponses queues to AddressTransportAdapter, mirroring the existing pendingMessages pattern.
• dispatchTokenTransfer / dispatchPaymentRequest / dispatchPaymentRequestResponse queue when handlers are empty.
• onTokenTransfer / onPaymentRequest / onPaymentRequestResponse drain on first subscribe.
• Replaced AddressTransportAdapter.fetchPendingEvents no-op (line 1925-1928) with delegation to this.mux.fetchPendingEvents() (defense-in-depth for relay disconnect/reconnect).

2. Root cause #2 — Outer transport "vacuous ACK" (FIXED)

NostrTransportProvider.handleTokenTransfer initialized let allDurable = true. Because PaymentsModule registers onTokenTransfer on the per-address adapter, not on the outer provider, the outer provider's transferHandlers is permanently empty. Empty for-loop ⇒ allDurable stays true ⇒ vacuous "durable" ACK ⇒ updateLastEventTimestamp advances lastEventTs to alice's created_at without anybody having processed the token. Explains why bob's stored since=1779443013 matched alice's created_at exactly in the original bug report.

Fix in transport/NostrTransportProvider.ts:1767: let allDurable = this.transferHandlers.size > 0;.

3. Tests

• Unit: tests/unit/transport/MultiAddressTransportMux.pending-queue-223.test.ts — 7 deterministic regression tests. Verified FAIL pre-fix (4 failed / 3 passed) and PASS post-fix.
• E2E: tests/e2e/cross-process-nostr-delivery-223.test.ts — Option-A in-process cross-process simulation against real testnet (RUN_UXF_E2E=1 gated). Fails on real testnet, see below.

What's STILL broken (next investigator picks up here)

Real-testnet run with full diagnostic (run id b63xkhrm1, branch HEAD):

[#223-DIAG alice pre-faucet]   expected TokenId.fromNameTag('xp223acppchk')=0000762b3dc7ee27...
[#223-DIAG alice pre-faucet]   local nametagToken.id=                       0000762b3dc7ee27...
[#223-DIAG alice pre-faucet]   match? expected==local: true              ✓ (PROXY math is correct)

[#223] Alice funded: 1000000000 USDU                                       ✓ (faucet → TXF receive works)
[#223] bob destroyed (offline for entire send)                             ✓
[#223] alice.send() → status=submitted id=804c94b3-…                       ✓ (UXF bundle published)
[#223] alice transfer:confirmed — event is on the relay                    ✓ (durably stored)
[#223] bob re-opened (simulated new process)                               ✓
[#223] bob poll: confirmed=0 unconfirmed=0 total=0 tokens=0                ✗ (×120 polls over 180s)
[#223] bob final: confirmed=0 unconfirmed=0 total=0 tokens=0               ✗

× expected 0 to be greater than or equal to 500000

No errors logged. receive() did not throw. The token simply never reaches bob's wallet.

The discriminating fact

Faucet → alice works. Alice → bob does NOT. The only structural difference is the wire payload shape:

Sender	Wire shape	Receiver works?
Faucet	TXF ({sourceToken, transferTx})	✓ Yes
Alice (senderUxf=true default)	UXF ({kind: 'uxf-cid', cid: '…'})	✗ No, cross-process
Alice (same-process e2e uxf-send-receive.test.ts)	UXF	✓ Yes

So this bug is UXF-specific AND cross-process-specific.

Hypotheses for the next investigator

Ranked by plausibility:

1. Recipient-side ingest worker pool (T.3.E) — auto-installed in _payments.initialize at modules/payments/PaymentsModule.ts:2258. On reload, maybe the worker pool's mid-flight processing state (which existed in bob v1) is required to complete the ingest. Cold-restart loses it.

2. IPFS CAR fetch from cold gateway pool — alice pins uxf-car to her IPFS gateway, publishes CID via Nostr. Bob v2 starts cold, tries to fetch the CID via his gateway pool. Even though preflightSkip(['ipfs', …]) reports ipfs healthy, the specific CID may not have propagated to bob's gateway yet. Likely fails silently inside a try/catch in the UXF resolver.

3. recipientContextStorage / finalizationQueueStorage (G3/G7) — wired at core/Sphere.ts:2648 BEFORE payments.initialize(). If the UXF receive flow expects a context entry to be created during bob v1's lifetime that wasn't persisted, bob v2's reload fails to find it.

4. Live-sub vs fetchPendingEvents routing divergence — fetchPendingEvents calls mux.handleEvent like the live sub, but maybe the UXF ingest gates on a different signal that only the live-sub path triggers.

How to repro the failure cheaply

git checkout fix/issue-223-cross-process-nostr-delivery
# Unit tests pass (queueing fix isolated correctness)
npx vitest run tests/unit/transport/MultiAddressTransportMux.pending-queue-223.test.ts
# E2E fails on real testnet — this is the surface area that needs more work
RUN_UXF_E2E=1 npm run test:e2e -- tests/e2e/cross-process-nostr-delivery-223.test.ts

Total e2e runtime ~4-5 minutes on healthy testnet (preflight 30s + faucet 60s + send-confirm + 180s receive-poll-then-fail).

Test methodology bug worth noting

Initial multi-scenario test setup used uppercase nametag prefixes (xpA-a-… / xpB-b-…). The SDK normalizes nametags to lowercase (NametagUtils.normalizeNametag), but the testnet faucet appears to compute its destination PROXY from the raw string. Result: alice's local ProxyAddress.fromTokenId(token.id) (lowercase hash) ≠ faucet's ProxyAddress.fromNameTag(rawString) (uppercase hash) → PROXY mismatch on finalize. Fixed by using all-lowercase prefixes xp223a/xp223b. The current branch's e2e test is correct. Worth a follow-up on the faucet / SDK to ensure both sides normalize identically — but it's not the #223 bug.

Open question for triage

Is the partial fix on this branch worth landing as-is (it correctly fixes the Mux adapter race and adds high-confidence unit-test coverage), or hold the branch until the UXF cross-process gap is also fixed? My read: land it (the Mux race is real and the unit tests prove the fix is sound), and open a new issue specifically for the UXF cross-process delivery gap so it gets dedicated attention. But this is your call.

cc'ing this to the next investigator: the bug is real, reproducible, and gated behind RUN_UXF_E2E=1. The branch contains everything you need to pick up exactly where this left off.

[vrogojin]

2026-05-23 testnet re-validation — bug still present

Re-ran manual-test-full-recovery.sh end-to-end after sphere-cli #21 landed (the CLI process.exit wrapper crash that was masking this gap — see sphere-cli/pull/22). The script now runs cleanly through §C without exiting on the CLI crash, which lets us see the SDK-side symptom directly.

peer2-alice's daemon never received Bob's payment DM.

• peer2-alice/.sphere-cli/daemon.log shows the daemon subscribed correctly: Subscribed events: transfer:incoming, transfer:confirmed, invoice:payment, invoice:covered and Daemon running. Waiting for events...
• After §C.2 (Bob pays alice's 11 UCT invoice over Nostr), peer2-alice/events.log is empty at the §C.3 60s wait timeout.
• peer2-bob/events.log DID receive its own outbound invoice:payment events for the same flow, so the relay + Nostr round-trip works in at least one direction. The asymmetry (live-subscribed peer2-alice doesn't see Bob's DM, but local-emitting peer2-bob does see its own outbound) matches the symptom in the issue body.
• Run preserved with KEEP=1 at /home/vrogojin/sphere-full-test-manual/ (peer1, peer2-alice, peer2-bob with daemon.log, events.log, snapshots).
• Full log: /tmp/manual-cli-full-run.log (script exit code 1 from §C.4's invoice status — see "Stacking bugs" below).

INV ID for cross-reference: 000038e7c929759dc4d5994c2a6fc5190563da713f70f8554fd57c182b078e3ad678.

Cross-link: #226

#226 ("Invoice delivery: payer can't discover invoices") was the original invoice-side surface of the same manual-test breakage and was partially fixed by sphere-cli PR #18 (the sphere invoice deliver subcommand). But that fix relies on the payer being reachable via Nostr DM at delivery time. This issue (#223) is the load-bearing primitive underneath — if Nostr cross-process delivery isn't reliable, then #226's DM-based delivery is also unreliable in the same way (just routed to a different recipient address).

Today's re-run effectively isolates #226 from #223: peer1-alice's sphere invoice deliver to Bob completed at §C.1b (Bob did receive the DM and could pay). But peer2-alice (alice's second device, same identity, subscribed to alice's transport pubkey) does not see Bob's reciprocal payment DM. So #226's invoice deliver does work when the recipient's subscription happens to catch the event, while #223's underlying gap continues to cause silent drops for the recipient device that started its subscription after the publish.

Stacking bugs in §C.4 (for context — most are NOT this issue)

Today's investigation found §C.4's symptom is actually FOUR stacking bugs. Listing here only for situational awareness; only Bug 1 is in scope for #223 — the others have their own follow-ups or are sphere-cli-side:

1. This issue (Cross-process Nostr token transfer is broken — CLI sender→exit→receiver→start loses events; no e2e coverage #223) — Nostr cross-process delivery gap.
2. CLI on deprecated IpfsStorageProvider (last-writer-wins IPNS clobbers cross-device data even before the accounting layer sees it). Architectural — sphere-cli needs to migrate to createNodeProfileProviders.
3. AccountingModule.invoiceTermsCache never refreshes from sync:completed — even if (2) were fixed, pulled invoice tokens wouldn't appear in getInvoices(). Independent sphere-sdk issue.
4. legacy-cli.ts:4028 invoice-status uses ensureSync(sphere, 'nostr') instead of 'full' — doesn't trigger IPFS pull at all. sphere-cli-side trivial fix.

Bugs 2–4 will each need their own tracking; flagging them so we don't accidentally fold them into the scope of #223. Full diagnosis with reproducer details preserved internally.

[vrogojin]

2026-05-23 follow-up — bug isolated to manual-test script, not the SDK

After diagnosing the preserved artifacts from this morning's run (`/home/vrogojin/sphere-full-test-manual/`) and re-running on a fresh testnet workspace with the corrected script, the §C.2 "peer2-alice's daemon never received Bob's payment DM" failure was a script bug, not an SDK regression. PR #225's cross-process Nostr delivery fix is sound — the surface area is verified working on `integration/all-fixes` HEAD.

Diagnosis of yesterday's symptom

peer1's `history.json` for Bob shows:
```
"recipientPubkey": "ab0faf6f…" (Bob's own pubkey)
"recipientNametag": "bob-7188094b" (Bob himself)
"recipientAddress": "DIRECT://000047951443…e507d" (Bob's own DIRECT)
```

Bob sent 11 UCT to his own address. The kind:31113 Nostr event's `#p` tag was Bob's transport pubkey, so peer2-bob's pubkey-filtered subscription saw it and peer2-alice's didn't. That's the asymmetry mentioned in the comment — but it was caused by the invoice's destination, not by a cross-process delivery gap.

Root cause: the manual-test script called `sphere invoice create --target "@$BOB_TAG"`. Per `modules/accounting/types.ts`, `--target` is the receiver of funds, so this made Bob the receiver of his own invoice. When Bob ran `invoice pay`, his wallet correctly sent 11 UCT to the invoice's target — himself.

Verification — corrected script works

• `tests/e2e/cross-process-nostr-delivery-223.test.ts` on `integration/all-fixes` HEAD: PASS (Bob received 500000 USDU after cross-process restart on live testnet, ~87s test runtime).
• Corrected `manual-test-full-recovery.sh` re-run against testnet: `peer2-alice/events.log` populated with 12 events (8 transfer:incoming + 4 transfer:confirmed) from sender `bob-XXX`, amount `11000000` UCT, recipient = alice's DIRECT address. The 2026-05-23 broken run had ZERO events in this file.
• `peer2-bob/events.log` correctly does NOT exist in the re-run — no Nostr event matches Bob's pubkey filter (his second device updates via IPFS Profile-pointer sync, exercised in §C.4's balance check).

PR

#229 `docs(manual-test)(#223): fix §C invoice direction — alice is receiver` lands the three-line script correction (`--target @bob` → `@alice`, add `--to @bob` to deliver, drop the misleading peer2-bob "transfer:" wait) plus comment refinements. No SDK changes.

Status on acceptance criteria

• Failing cross-process e2e test added — `cross-process-nostr-delivery-223.test.ts` (PR fix(payments)(#223): complete cross-process UXF delivery — cidFetchGateways + block-walk fetch #225)
• Root cause isolated — Mux adapter race + UXF block-walk fetch (PR fix(payments)(#223): complete cross-process UXF delivery — cidFetchGateways + block-walk fetch #225)
• Fix lands; same e2e test passes — verified today on `integration/all-fixes` HEAD
• [⚠️] Manual walkthrough runs cleanly — script fix in docs(manual-test)(#223): fix §C invoice direction — alice is receiver #229 makes the §C cross-process Nostr signal green. §C.4 and §D-§E assertions remain blocked by the three out-of-scope follow-ups already enumerated below.

Out-of-scope follow-ups (each warrants its own issue)

Today's re-run hit these on §C.4 / §D / §E (independent of the §C.2 script bug fix):

2. sphere-cli still uses the deprecated `IpfsStorageProvider` — Profile-pointer sync may not propagate cross-device.
3. `AccountingModule.invoiceTermsCache` never refreshes from `sync:completed` — peer2-alice's `invoice status $INV` returned "No invoice found matching prefix" even though the payment landed via Nostr.
4. `legacy-cli.ts:4028` uses `ensureSync(sphere, 'nostr')` instead of `'full'` — `invoice status` doesn't trigger IPFS pull.

A separate tangential signal observed (worth its own issue):

5. `PaymentsModule.finalizeStrandedReceivedToken` repeatedly throws `InvalidJsonStructureError` from `InclusionProof.fromJSON` on the stranded receive — non-fatal but noisy and may block the V6-recover worker from converging.

Recommendation

Land PR #229 and close this issue. Open separate issues for items 2-5 above; they all relate to the §C.4 / §D / §E manual-test assertions but are independent code paths from #223's cross-process Nostr delivery.

[vrogojin]

Follow-up tracking

Four follow-up issues filed for the §C.4 / §D / §E gaps surfaced by today's manual-test re-run (none in scope for the SDK fix that PR #225 landed):

Item	Repo	#	Effort
2	sphere-cli	#23 — Migrate from deprecated IpfsStorageProvider to createNodeProfileProviders	Multi-day architectural
3	sphere-sdk	#230 — AccountingModule.invoiceTermsCache stale on cross-device sync	Medium (subscribe to sync:completed)
4	sphere-cli	#24 — invoice-status uses ensureSync('nostr'), should be 'full'	Trivial (one-line)
5	sphere-sdk	#231 — PaymentsModule.finalizeStrandedReceivedToken retry-loops on InvalidJsonStructureError	Medium (classify as permanent failure)