|
| 1 | +# Asset blob offload — design |
| 2 | + |
| 3 | +*Design document, July 2026. Status: **proposed** — nothing built. Companion |
| 4 | +to `relay-design.md` ("Wire efficiency") and `collab-design.md`. The immediate |
| 5 | +motivation is a live bug: media cannot be added during a collaboration session |
| 6 | +at all.* |
| 7 | + |
| 8 | +## The problem |
| 9 | + |
| 10 | +`crdt.ts` syncs an asset as a **whole-value LWW register** — one `set` op whose |
| 11 | +value is the entire base64 data URI. That runs into a hard ceiling: |
| 12 | + |
| 13 | +- An asset costs **~1.78× its binary size** on the wire (base64 twice — once |
| 14 | + into the data URI, once over the ciphertext). |
| 15 | +- A Durable Object storage value caps near **2 MB** (measured, not documented: |
| 16 | + 2 MB stores, 2.5 MB throws inside `storage.put()`). |
| 17 | +- So the practical ceiling is **~1.05 MB of binary**, against a |
| 18 | + `MEDIA_EMBED_BUDGET` of 8 MB. |
| 19 | + |
| 20 | +Raising the frame limit does not help — the Cloudflare WebSocket limit is now |
| 21 | +32 MiB, but *storage* is the binding constraint. No constant fixes this. |
| 22 | + |
| 23 | +Two further consequences, both fixed by this design: |
| 24 | + |
| 25 | +- **Snapshots carry every asset** (`session.snapshot()` serialises the whole |
| 26 | + document), so a media-heavy deck can never snapshot — and since the relay |
| 27 | + only prunes covered ops when a snapshot lands, its op log grows unbounded |
| 28 | + for the room's whole 30-day life. |
| 29 | +- **No dedupe.** The same image referenced twice is stored twice; re-adding it |
| 30 | + writes it again. |
| 31 | + |
| 32 | +## The constraint that shapes everything |
| 33 | + |
| 34 | +**The file must stay self-contained.** A `.bento.html` carries its assets as |
| 35 | +data URIs in `doc.assets` — that is the product. So this is a **transport |
| 36 | +change, not a format change**. `doc.assets` keeps holding data URIs; only the |
| 37 | +way an asset *travels between peers* changes. A receiving peer fetches the |
| 38 | +blob, decrypts, and materialises it back into `doc.assets` exactly as if it |
| 39 | +had arrived inline. |
| 40 | + |
| 41 | +Get this wrong and the single-file invariant breaks (`PLATFORM.md` §1). |
| 42 | + |
| 43 | +## Model |
| 44 | + |
| 45 | +**The op stops carrying bytes.** Above a threshold (~64 KB — below that, |
| 46 | +inlining is cheaper than the round trip) the differ emits a reference instead |
| 47 | +of the value: |
| 48 | + |
| 49 | +``` |
| 50 | +set assets.<k> = { __blob: "<storage key>", hash, size, mime } |
| 51 | +``` |
| 52 | + |
| 53 | +A couple of hundred bytes instead of megabytes. |
| 54 | + |
| 55 | +**Content-addressed, with a privacy wrinkle.** The natural id is |
| 56 | +`sha256(plaintext)`, which gives dedupe for free — but if the relay sees that |
| 57 | +id, it can tell two different rooms hold the same image. Use |
| 58 | +**`HMAC(roomKey, sha256(plaintext))`** as the storage key instead: |
| 59 | +deterministic *within* a room so dedupe works, opaque *across* rooms so |
| 60 | +nothing correlates. Cheap now, awkward to retrofit. |
| 61 | + |
| 62 | +**Blobs move over HTTP, not the WebSocket.** `PUT /b/<room>/<key>` and |
| 63 | +`GET /b/<room>/<key>`, backed by R2. This sidesteps the 2 MB storage-value |
| 64 | +limit entirely, gives resumable multipart upload for free, and R2 charges no |
| 65 | +egress. It is also exactly the vault dead-drop primitive — build it once |
| 66 | +(`vault-design.md`). |
| 67 | + |
| 68 | +**Upload before you emit.** An op must never reference a blob that is not yet |
| 69 | +fetchable. Belt and braces: a peer that receives a reference to an unknown |
| 70 | +blob fetches on demand and renders a placeholder meanwhile, which also covers |
| 71 | +replay and late joiners. |
| 72 | + |
| 73 | +**Encryption** uses the existing room key, same as frames. The relay stores |
| 74 | +ciphertext and can no more read a blob than a frame. |
| 75 | + |
| 76 | +## The risky seam |
| 77 | + |
| 78 | +This makes an op's meaning depend on an **external fetch**: applying an op |
| 79 | +becomes async and failable, which breaks the CRDT's "ops are pure data" |
| 80 | +property. |
| 81 | + |
| 82 | +Mitigations: |
| 83 | + |
| 84 | +- There is precedent in the engine — the `pending` buffer already defers ops |
| 85 | + whose target node is unknown, so *deferred until resolvable* is an existing |
| 86 | + shape rather than a new concept. |
| 87 | +- Keep `crdt.ts` changes minimal: the differ and apply path should treat |
| 88 | + `{ __blob }` as an **opaque value**, with a separate resolution layer above |
| 89 | + them doing the fetching and substitution. The CRDT should not know what a |
| 90 | + blob is. |
| 91 | +- Extend `scripts/test-sync.ts` to cover blob-reference ops before merging |
| 92 | + anything. (Note it currently only generates short strings — which is why it |
| 93 | + missed the large-text stack overflow fixed in #47. Assume it will miss this |
| 94 | + class too unless extended.) |
| 95 | + |
| 96 | +## Failure modes, stated plainly |
| 97 | + |
| 98 | +- **Blob upload fails, op already sent** — prevented by upload-before-emit, |
| 99 | + but a crash between the two is possible. The receiver renders a placeholder |
| 100 | + and the asset stays missing until the sender re-uploads on next connect. |
| 101 | +- **Blob expired, peer never fetched it** — a peer holds an element |
| 102 | + referencing an asset nobody can produce. This is permanent divergence and it |
| 103 | + needs a deliberate answer: render a placeholder with an honest "asset |
| 104 | + unavailable" state, never a silent broken image. TTL should match the room's |
| 105 | + 30-day expiry, which is long enough that any peer who opened the document |
| 106 | + and saved has the bytes in their own file. |
| 107 | +- **Storage quota reached** — refuse loudly with a code, exactly as frames now |
| 108 | + do. Never drop silently; that class of bug cost us a permanent resend loop. |
| 109 | + |
| 110 | +## Sequence |
| 111 | + |
| 112 | +1. **Relay blob endpoints + R2.** `PUT`/`GET`, per-room namespace, TTL matched |
| 113 | + to room expiry, size and quota caps from the start. Independently testable |
| 114 | + with no client changes. |
| 115 | +2. **Client blob layer.** Content hashing, HMAC key derivation, an |
| 116 | + IndexedDB-backed cache, fetch-on-demand with placeholder rendering. |
| 117 | +3. **Wire it into the differ** above the threshold; extend the convergence rig |
| 118 | + in the same PR. |
| 119 | +4. **Chunked/resumable upload** for genuinely large single blobs. |
| 120 | + |
| 121 | +Steps 1 and 2 touch neither `crdt.ts` nor the document format, so they can |
| 122 | +proceed while other sync work lands. |
| 123 | + |
| 124 | +## Verify before building — do not trust these assumptions |
| 125 | + |
| 126 | +Today's session produced three confident conclusions that measurement |
| 127 | +overturned (the 1 MB frame limit, "chunking is unnecessary", and the RGA |
| 128 | +stack overflow). Treat the following the same way: |
| 129 | + |
| 130 | +- **Can a Worker stream a multi-MB blob to/from R2 without buffering it in |
| 131 | + memory?** The 128 MB Worker memory limit is exactly the kind of constraint |
| 132 | + that is not in the docs you would reason from. |
| 133 | +- **What is R2's actual behaviour under the free/paid plan** for many small |
| 134 | + objects, and what do per-operation costs look like at realistic churn? |
| 135 | +- **Does `crypto.subtle` streaming decryption work** for a blob larger than |
| 136 | + memory, or must it be chunked for that reason alone (independent of upload |
| 137 | + resumability)? |
| 138 | + |
| 139 | +Answer these with a probe against a real deployment before the design hardens. |
0 commit comments