diff --git a/recipes/synthesis-capture/DEPENDENCIES.md b/recipes/synthesis-capture/DEPENDENCIES.md new file mode 100644 index 000000000..a6accdb13 --- /dev/null +++ b/recipes/synthesis-capture/DEPENDENCIES.md @@ -0,0 +1,96 @@ +# Synthesis Capture — Dependencies & Known Limitations + +This recipe has two unresolved couplings to the rest of Open Brain that a +contributor installing against stock `origin/main` should understand before +deploying. + +## 1. Sibling recipe: `provenance-chains` (pending) + +### What's blocked + +The stock `upsert_thought` RPC shipped in +[`docs/01-getting-started.md`](../../docs/01-getting-started.md) reads +**only** `p_payload.metadata` when inserting into `public.thoughts`: + +```sql +INSERT INTO thoughts (content, content_fingerprint, metadata) +VALUES (p_content, v_fingerprint, COALESCE(p_payload->'metadata', '{}'::jsonb)) +``` + +Top-level keys on `p_payload` — including `source_type`, +`derivation_layer`, `derivation_method`, and `derived_from` — are silently +dropped. The sibling `provenance-chains` recipe (branch +`contrib/alanshurafa/provenance-chains`, not yet merged) ships: + +1. Three new columns on `public.thoughts`: + `derivation_layer` (text), `derivation_method` (text), `derived_from` (jsonb). +2. An updated `upsert_thought` RPC that reads and persists those top-level fields. + +### Interim mitigation (already applied in this recipe) + +Both handlers mirror the provenance fields into +`metadata.provenance.{source_type, derivation_layer, derivation_method, derived_from}` +in addition to passing them at the top level of `p_payload`. This means: + +- **On the patched RPC (after `provenance-chains` lands):** top-level fields + populate their dedicated columns; the metadata mirror is redundant but + harmless. +- **On the stock RPC (current `main`):** top-level fields are dropped but + the metadata mirror survives. Consumers can reconstruct provenance by + reading `metadata->'provenance'`. + +Search for `TODO(synthesis-capture)` in `mcp-tool-handler.ts` and +`rest-endpoint.ts` for the exact locations to clean up once the patched +RPC is in place. + +### Caveats while `provenance-chains` is unmerged + +- The anti-loop check (`source_type = 'synthesis'`) reads the top-level + column in both handlers. On stock RPC, no synthesis row ever has that + column populated, so the check will never reject a + synthesis-of-synthesis. Treat this as a best-effort guard, not a + guarantee, until the patched RPC is deployed. +- The `derivation_method = 'synthesis'` SQL filter in README "How to + Verify" will return zero rows on stock RPC. Substitute + `metadata->'provenance'->>'derivation_method' = 'synthesis'` in the + meantime. + +## 2. Base tools don't expose row IDs + +### What's blocked + +`capture_synthesis` requires `source_thought_ids: [id, id, id, ...]`. In a +typical MCP session, an AI client discovers those IDs by first calling +`search_thoughts` or `list_thoughts`. The stock versions of those tools +(see `server/index.ts` around lines 113 and 202 on `origin/main`) return +formatted text for human consumption and **never include the raw row IDs +in the tool response**. + +The result: a model following the README example ("search my brain, then +capture your summary as a synthesis with provenance") cannot actually +produce the required input list on its own. It either guesses IDs and +fails the existence check, or gives up. + +### Workarounds (until a base update lands) + +- **Manual ID injection:** the human user pastes IDs into the prompt. + ExoCortex dashboards, direct SQL, or a custom recipe can surface IDs for + copy/paste. +- **Custom read tool:** deploy a variant of `search_thoughts` that returns + structured JSON including `id`, then reference it instead. The + sibling `panning-for-gold` skill contains one such pattern; this recipe + does not ship its own. +- **REST path:** the `POST /synthesis` endpoint is unaffected — callers + pulling IDs from scripts or dashboards already have them. + +### TODO + +Once the base MCP tools expose IDs (tracked as a follow-up against +`server/index.ts`, no ticket yet), this section can be retired. Until +then, expect to see failures of the form +`parent thoughts not found: ...` when a model tries to invent IDs from +thin air — it's not a recipe bug, it's a base-tool limitation. + +--- + +_Last updated: 2026-04-17_ diff --git a/recipes/synthesis-capture/README.md b/recipes/synthesis-capture/README.md new file mode 100644 index 000000000..9a2efaa5d --- /dev/null +++ b/recipes/synthesis-capture/README.md @@ -0,0 +1,226 @@ +# Synthesis Capture + +> Let your AI save its own synthesis of multiple thoughts as a new thought — the "Query-as-Ingest" pattern. Your brain compounds instead of re-deriving. + +## What It Does + +Adds two entry points for capturing a synthesis with provenance: + +- `capture_synthesis` — MCP tool your AI client can call +- `POST /synthesis` — REST endpoint for scripts, webhooks, and non-MCP clients + +When your AI answers a complex question by combining multiple atomic thoughts, this recipe lets it save the answer itself as a new thought whose `derived_from` column points back to every source. Next time you ask a similar question, the synthesis is already there — fewer tokens burned, fewer roundtrips, and a clear audit trail from belief to evidence. This is what Andrej Karpathy calls "Query-as-Ingest": queries stop being read-only and start compounding the brain. + +## Prerequisites + +- Working Open Brain setup ([Getting Started guide](../../docs/01-getting-started.md)) +- **The `provenance-chains` sibling recipe applied (recommended)** — this recipe depends on two things that ship together in that recipe: + 1. Three new columns on `public.thoughts`: `derivation_layer`, `derivation_method`, `derived_from`. + 2. An updated `upsert_thought` RPC that reads those three fields from the top level of `p_payload` (not just from `p_payload.metadata`). + + **On the stock RPC (no `provenance-chains` yet):** inserts still succeed and provenance is preserved in `metadata.provenance.{source_type,derivation_layer,derivation_method,derived_from}` as a mirrored fallback. The top-level provenance columns populated by the patched RPC will be empty, but you can reconstruct the chain from metadata. The anti-loop safety guard (synthesis-of-synthesis rejection) reads the top-level column and is therefore best-effort on stock RPC — see [`DEPENDENCIES.md`](./DEPENDENCIES.md) for the full matrix. + + See [Step 1](#step-1-confirm-the-provenance-columns-exist) for a quick verification query and [Known Limitations](#known-limitations) for the broader dependency graph. +- An `open-brain-mcp` Edge Function deployed from [server/index.ts](../../server/index.ts) — this recipe adds a second tool alongside `capture_thought`. +- An `open-brain-rest` Edge Function deployed (for the REST half). If you only want the MCP tool, skip `rest-endpoint.ts`. +- Supabase CLI linked to your project (for redeploying after you add the handler code). + +> [!IMPORTANT] +> This recipe **only persists** a synthesis — it does not generate one. The caller (your AI client, your script) must produce the synthesis prose and pass it in. Keeping the LLM off the hot path keeps the capture path cheap, deterministic, and free of server-side API keys. + +> [!NOTE] +> **UUID vs BIGINT IDs.** The stock OB1 schema in the Getting Started guide uses `UUID` primary keys on `public.thoughts`. Some enhanced variants use `BIGINT`. Both handlers in this recipe accept either — source IDs are treated as opaque values and only compared by equality. If your install uses UUIDs, pass them as JSON strings (`"11111111-..."`), not numbers. + +## What's In This Folder + +| File | Purpose | +|------|---------| +| `mcp-tool-handler.ts` | `capture_synthesis` tool. Paste into your `open-brain-mcp` Edge Function. | +| `rest-endpoint.ts` | `POST /synthesis` handler. Paste into your `open-brain-rest` Edge Function. | +| `metadata.json` | OB1 contribution metadata. | +| `README.md` | This file. | + +## Safety Rules + +These are enforced identically in both the MCP tool and the REST endpoint. They exist to prevent the brain from compounding its own noise. + +| Rule | Why it exists | +|------|---------------| +| **At least 3 source thought IDs** | A "synthesis" of one or two thoughts is usually just a rewrite. Three is the smallest number where combining them genuinely produces new knowledge. | +| **All source IDs must exist in `public.thoughts`** | Silent dangling references poison provenance walks. We fail loud instead. | +| **No source may have `source_type = 'synthesis'`** | Forbids synthesis-of-synthesis. Otherwise the system could recursively compound its own derivations and drift arbitrarily far from real evidence. | +| **At least one source must have `derivation_layer = 'primary'`** | Guarantees every synthesis chain is ultimately rooted in an atomic, captured-from-reality thought — not just a pile of other derivations. | + +If any rule fails, the insert is rejected with a 400-class error. Nothing is written. + +--- + +## Setup + +### Step 1: Confirm the provenance columns exist + +Run this in your Supabase SQL editor: + +```sql +select column_name, data_type +from information_schema.columns +where table_schema = 'public' + and table_name = 'thoughts' + and column_name in ('derivation_layer', 'derivation_method', 'derived_from') +order by column_name; +``` + +You should see three rows: + +| column_name | data_type | +|-------------|-----------| +| `derivation_layer` | `text` | +| `derivation_method` | `text` | +| `derived_from` | `jsonb` | + +If any are missing, stop here and apply the `provenance-chains` recipe's migration first. + +Done when: All three columns appear in the result. + +--- + +### Step 2: Add the MCP tool handler + +Open your local copy of `server/index.ts` (the file you deployed as `open-brain-mcp`). Find the `capture_thought` `registerTool` call. Paste the entire contents of `mcp-tool-handler.ts` directly after it — the block is a single `server.registerTool(...)` call and needs the same scope (`server`, `supabase`, `z`, `getEmbedding`, `extractMetadata` already in scope from OB1's default `server/index.ts`). + +Done when: Your `server/index.ts` has both `capture_thought` and `capture_synthesis` tools registered, and `deno check server/index.ts` (or your normal type check) passes. + +--- + +### Step 3: Add the REST endpoint (optional, only if you run `open-brain-rest`) + +Open your `open-brain-rest` function. Paste the contents of `rest-endpoint.ts` at the bottom of the file (it exports `handleCaptureSynthesis`). Then wire the route into your router: + +**For a `Deno.serve` / `URL`-based router:** + +```ts +if (path === "/synthesis" && req.method === "POST") { + return await handleCaptureSynthesis(req); +} +``` + +**For a Hono-based router:** + +```ts +app.post("/synthesis", (c) => handleCaptureSynthesis(c.req.raw)); +``` + +Done when: Sending `OPTIONS /synthesis` returns a CORS-allowed response and `POST /synthesis` with an empty body returns `{"error": "Invalid JSON in request body"}` (proves the route is wired but validation still works). + +--- + +### Step 4: Redeploy + +```bash +supabase functions deploy open-brain-mcp +supabase functions deploy open-brain-rest # only if you added the REST endpoint +``` + +Done when: Both deploys report success and the function log shows no startup errors. + +--- + +## Usage + +### Example 1: MCP call from Claude Desktop + +Ask Claude (in a conversation connected to your Open Brain MCP server): + +> "Search my brain for my thoughts on Postgres pgvector performance. Summarize the key takeaways, then capture your summary as a synthesis with provenance back to the source IDs." + +Claude will (a) call `search_thoughts`, (b) compose the synthesis, (c) call `capture_synthesis` with the source IDs it used. You should see something like: + +``` +Captured synthesis #40221 from 5 source thoughts. Future queries on this topic can reuse it directly. +``` + +### Example 2: REST call via curl + +```bash +curl -X POST https://.supabase.co/functions/v1/open-brain-rest/synthesis \ + -H "Content-Type: application/json" \ + -H "x-brain-key: $OB1_BRAIN_KEY" \ + -d '{ + "content": "pgvector HNSW index builds are slow the first time but consistently outperform IVFFlat for recall@10 on embedding tables >1M rows.", + "source_thought_ids": [12033, 14288, 15901, 17442], + "question": "Is HNSW worth the build cost vs IVFFlat?", + "topics": ["pgvector", "vector-search", "postgres"] + }' +``` + +> [!TIP] +> If your install uses UUID IDs (the stock Getting Started schema), pass them as quoted strings instead: `"source_thought_ids": ["b5a...","9f1...","e77..."]`. + +Expected response: + +```json +{ + "thought_id": 40221, + "source_count": 4, + "message": "Captured synthesis #40221 from 4 source thoughts" +} +``` + +Replay the curl with the same content and source IDs and you'll get the same `thought_id` back — `upsert_thought` de-duplicates by content fingerprint, so syntheses are idempotent. + +--- + +## How to Verify It's Working + +Run this after capturing your first synthesis: + +```sql +select id, source_type, derivation_layer, derivation_method, derived_from, left(content, 80) as preview +from public.thoughts +where derivation_method = 'synthesis' +order by id desc +limit 5; +``` + +You should see your new synthesis with `derivation_layer = 'derived'`, `derivation_method = 'synthesis'`, and `derived_from` populated with the source IDs you passed in. + +To walk the provenance chain, use the `trace_provenance` MCP tool or `GET /thought/:id/provenance` endpoint from the `provenance-chains` recipe. + +## Troubleshooting + +### `column "derivation_layer" of relation "thoughts" does not exist` + +The `provenance-chains` migration hasn't been applied to this database. Apply it before using this recipe. Re-run the verification query in Step 1 until all three columns are present. + +### `Error: source_thought_ids must include at least 3 distinct IDs` + +The caller passed fewer than 3 unique IDs. Duplicates are collapsed before the count check, so passing `[101, 101, 102]` reads as 2 distinct IDs and fails. Pass at least 3 genuinely different source thoughts. + +### `Error: all source thoughts are derived; at least one must be a primary (atomic) thought` + +Every ID you passed points to a row whose `derivation_layer = 'derived'`. That is intentionally blocked to stop recursive synthesis chains. Either include at least one primary thought in the source set, or reconsider whether the inputs you have are really strong enough for a synthesis. + +### `Error: source thoughts [...] are themselves syntheses` + +One or more source IDs already have `source_type = 'synthesis'`. Anti-loop rule: syntheses can only be built from non-synthesis thoughts. Trace those synthesis IDs back to their own `derived_from` and include the underlying primary thoughts instead. + +### `Error: upsert_thought returned no thought ID` + +Your `upsert_thought` RPC is returning an unexpected shape. This recipe expects `{ id: }` in the result. If you are on a forked RPC, adapt the `(upsertResult as { id?: number })` destructure at the bottom of both handlers to match what your RPC returns. + +### MCP tool registered but Claude can't see it + +After redeploying, disconnect and reconnect the Open Brain connector in Claude Desktop (Settings → Connectors → disable → re-enable). MCP tool lists are cached on the client side and a reconnect forces a fresh `tools/list` roundtrip. + +--- + +## Known Limitations + +See [`DEPENDENCIES.md`](./DEPENDENCIES.md) for full detail. Summary: + +1. **Stock `upsert_thought` RPC drops top-level provenance fields.** Until the sibling `provenance-chains` recipe lands with its patched RPC, this recipe mirrors provenance into `metadata.provenance.*` so the data is durable — but the top-level columns (`source_type`, `derivation_layer`, `derivation_method`, `derived_from`) will be unpopulated on stock installs. The anti-loop safety guard reads the top-level column and is therefore best-effort until `provenance-chains` lands. +2. **Stock `search_thoughts` / `list_thoughts` don't expose row IDs.** An AI client following the "search then synthesize" flow in Example 1 cannot read the required `source_thought_ids` from the standard tools — they only return formatted text. Workaround: pass IDs manually from a dashboard or SQL query, or deploy a variant read tool that returns structured JSON. A base update that exposes IDs is tracked as a follow-up; no timeline yet. +3. **Input caps.** `content` is capped at 50KB, `source_thought_ids` at 50 items, `question` at 2000 chars, `topics` at 20 entries of ≤100 chars each, `tags` at 20 entries of ≤50 chars each, and `metadata` at 50 keys with a fully-merged size ≤10KB. Over-cap requests return HTTP 413 (REST) or an `isError: true` MCP response with a field-specific message. Adjust in both `mcp-tool-handler.ts` (Zod schema + post-merge size guard) and `rest-endpoint.ts` (imperative checks) together if your use case needs higher bounds. +4. **Embedding soft-fail.** If the embedding patch write fails after the row is saved, both MCP and REST paths return success with a warning message (`embedding_error`). The thought is durable — callers can call `capture_synthesis` again (it is idempotent via content fingerprint); if the problem persists, contact your admin or check the Open Brain embedding service. + +Search the source files for `TODO(synthesis-capture):` to locate the exact lines where these limitations originate. diff --git a/recipes/synthesis-capture/mcp-tool-handler.ts b/recipes/synthesis-capture/mcp-tool-handler.ts new file mode 100644 index 000000000..9d8d3cc05 --- /dev/null +++ b/recipes/synthesis-capture/mcp-tool-handler.ts @@ -0,0 +1,317 @@ +/** + * capture_synthesis — MCP tool handler for Open Brain + * + * Paste this inside your open-brain-mcp Edge Function's server setup, next to + * the existing `capture_thought` registerTool call (see OB1's server/index.ts). + * + * What it does + * ------------ + * Lets an AI client save its own synthesis of multiple source thoughts as a + * NEW thought, with `derived_from` pointing back to every source ID. The + * "Query-as-Ingest" pattern: once a complex question has been answered by + * combining atomic thoughts, the answer becomes reusable knowledge instead of + * something the model has to re-derive every time. + * + * Requirements + * ------------ + * This handler writes to three provenance columns on `public.thoughts`: + * - derivation_layer (text: 'primary' | 'derived') + * - derivation_method (text: 'synthesis') + * - derived_from (jsonb: array of parent thought IDs) + * + * Those columns are added by the `provenance-chains` recipe's schema + * migration. This recipe will NOT work without that migration applied. + * + * Safety rules enforced here + * -------------------------- + * 1. At least 3 source thought IDs required (Zod `.min(3)`). + * 2. All source IDs must resolve to existing rows in `public.thoughts`. + * 3. At least one source must have `derivation_layer = 'primary'` — forbids + * pure synthesis-of-synthesis chains that would compound noise. + * 4. Rejects sources that are themselves `source_type = 'synthesis'` so the + * system cannot loop on its own derived output. + * + * Assumes in scope + * ---------------- + * - `server` : your McpServer instance (from @modelcontextprotocol/sdk) + * - `supabase` : your Supabase service-role client + * - `z` : zod + * - `getEmbedding(text)` : your existing embedding helper (OB1 default) + * - `extractMetadata(text)`: your existing metadata helper (OB1 default) + * + * If your `open-brain-mcp` already wraps these differently, adapt the two + * helper calls below — the rest of the logic is independent. + */ + +server.registerTool( + "capture_synthesis", + { + title: "Capture Synthesis", + description: + "Capture a derived-synthesis thought from 3+ source thoughts (Query-as-Ingest). At least one source must be a primary (atomic) thought. No source may already be a synthesis. Use this after answering a complex question so the answer itself becomes reusable knowledge with provenance back to the sources.", + inputSchema: { + // Size cap rationale: 50KB of UTF-8 covers a very long synthesis + // (~10k words) without leaving the endpoint wide open for DoS or + // accidental prompt-injection payload floods. Adjust upward only if + // you actually need longer syntheses — the embedding and DB write + // costs scale with content length. + content: z + .string() + .min(1) + .max(50_000, "content must be 50KB or less") + .describe("The synthesized answer/prose to save as a new thought (max 50KB)."), + // Accepts numeric IDs (BIGINT installs) or string IDs (UUID installs). + // The handler below treats IDs as opaque and only compares by equality. + // Cap at 50 to keep the `.in("id", ...)` query plan sane — a real + // synthesis rarely cites more than a dozen sources. + source_thought_ids: z + .array(z.union([z.number().int().positive(), z.string().min(1)])) + .min(3) + .max(50, "source_thought_ids must be 50 or fewer items") + .describe("Parent thought IDs the synthesis was derived from (minimum 3, maximum 50). Accepts integers or UUID strings depending on your thoughts.id type."), + question: z + .string() + .max(2_000, "question must be 2000 chars or less") + .optional() + .describe("Optional: the original question that prompted the synthesis."), + // Per-item caps match the REST path so both surfaces reject identical + // payloads. Adjust both handlers together if you need higher bounds. + topics: z.array(z.string().max(100, "topics entry exceeds 100 character limit")).max(20, "topics exceeds 20 item limit").optional(), + tags: z.array(z.string().max(50, "tags entry exceeds 50 character limit")).max(20, "tags exceeds 20 item limit").optional(), + }, + }, + async ({ content, source_thought_ids, question, topics, tags }) => { + try { + const trimmed = String(content ?? "").trim(); + if (!trimmed) { + return { + content: [{ type: "text" as const, text: "Error: content is required" }], + isError: true, + }; + } + + // De-dupe IDs before we hit the database — callers sometimes repeat. + const sourceIds = Array.from(new Set(source_thought_ids)); + if (sourceIds.length < 3) { + return { + content: [ + { + type: "text" as const, + text: "Error: source_thought_ids must include at least 3 distinct IDs", + }, + ], + isError: true, + }; + } + + // ── Safety check 1: all sources must exist ────────────────────────── + const { data: parents, error: parentsError } = await supabase + .from("thoughts") + .select("id, derivation_layer, source_type") + .in("id", sourceIds); + + if (parentsError) { + return { + content: [ + { + type: "text" as const, + text: `Error: parent lookup failed: ${parentsError.message}`, + }, + ], + isError: true, + }; + } + + // `id` is typed broadly because OB1 installs vary — the stock base + // schema in docs/01-getting-started.md uses UUID, while the enhanced / + // provenance-chains variants may use BIGINT. Both compare via Set.has() + // on the original value, so either shape works here. + const parentRows = (parents ?? []) as Array<{ + id: number | string; + derivation_layer: string | null; + source_type: string | null; + }>; + const foundIds = new Set(parentRows.map((p) => p.id)); + const missing = sourceIds.filter((id) => !foundIds.has(id)); + if (missing.length > 0) { + return { + content: [ + { + type: "text" as const, + text: `Error: parent thoughts not found: ${missing.join(", ")}`, + }, + ], + isError: true, + }; + } + + // ── Safety check 2: reject synthesis-of-synthesis ─────────────────── + const synthSources = parentRows.filter((p) => p.source_type === "synthesis"); + if (synthSources.length > 0) { + return { + content: [ + { + type: "text" as const, + text: + "Error: source thoughts [" + + synthSources.map((p) => p.id).join(", ") + + "] are themselves syntheses. Synthesis-of-synthesis is forbidden — pick primary (atomic) thoughts instead.", + }, + ], + isError: true, + }; + } + + // ── Safety check 3: at least one primary parent ───────────────────── + const anyPrimary = parentRows.some((p) => p.derivation_layer === "primary"); + if (!anyPrimary) { + return { + content: [ + { + type: "text" as const, + text: + "Error: all source thoughts are derived; at least one must be a primary (atomic) thought to prevent self-referential synthesis chains.", + }, + ], + isError: true, + }; + } + + // ── Build the synthesis row ───────────────────────────────────────── + // We reuse the OB1 pattern from capture_thought: embedding + metadata in + // parallel, upsert_thought RPC, then patch embedding. + const [embedding, autoMetadata] = await Promise.all([ + getEmbedding(trimmed), + extractMetadata(trimmed), + ]); + + const mergedMetadata: Record = { + ...(autoMetadata as Record), + source: "mcp_synthesis", + }; + if (question) mergedMetadata.question = String(question); + if (Array.isArray(topics) && topics.length > 0) { + mergedMetadata.topics = topics; + } + if (Array.isArray(tags) && tags.length > 0) { + mergedMetadata.tags = tags; + } + + // Belt-and-suspenders: mirror provenance fields into metadata so they + // survive the stock `upsert_thought` RPC, which only persists + // `p_payload.metadata` and silently drops top-level `p_payload` keys. + // TODO(synthesis-capture): once the sibling `provenance-chains` recipe + // lands on main with an updated RPC that reads top-level + // `source_type`, `derivation_layer`, `derivation_method`, and + // `derived_from`, this metadata mirror becomes redundant and can be + // removed. Until then, this is the ONLY code path that guarantees + // provenance lands somewhere queryable on stock installs. + // See DEPENDENCIES.md for the full rationale. + mergedMetadata.provenance = { + source_type: "synthesis", + derivation_layer: "derived", + derivation_method: "synthesis", + derived_from: sourceIds, + }; + + // Final size guard on the fully-merged metadata object, mirroring + // the REST path so both surfaces reject identical over-sized payloads. + // Checked AFTER merge (including provenance stamping) so callers cannot + // bypass the cap by shrinking individual fields. + const mergedSize = JSON.stringify(mergedMetadata).length; + if (mergedSize > 10_240) { + return { + content: [ + { + type: "text" as const, + text: `Error: metadata exceeds 10KB limit (got ${mergedSize} bytes after merge)`, + }, + ], + isError: true, + }; + } + + const { data: upsertResult, error: upsertError } = await supabase.rpc( + "upsert_thought", + { + p_content: trimmed, + p_payload: { + // Top-level provenance: works on the patched RPC from the + // sibling provenance-chains recipe. + source_type: "synthesis", + metadata: mergedMetadata, + derivation_layer: "derived", + derivation_method: "synthesis", + derived_from: sourceIds, + }, + }, + ); + + if (upsertError) { + return { + content: [ + { + type: "text" as const, + text: `Error: synthesis capture failed: ${upsertError.message}`, + }, + ], + isError: true, + }; + } + + const thoughtId = (upsertResult as { id?: number | string } | null)?.id; + if (!thoughtId) { + return { + content: [ + { + type: "text" as const, + text: "Error: upsert_thought returned no thought ID", + }, + ], + isError: true, + }; + } + + // Patch the embedding in a second write to match OB1's existing pattern. + const { error: embError } = await supabase + .from("thoughts") + .update({ embedding }) + .eq("id", thoughtId); + + if (embError) { + // Soft-fail: the thought row is durable; only the embedding failed. + // We return isError: false so the caller does not retry the whole + // capture (which would be idempotent via fingerprint anyway, but + // is wasteful). This matches REST-side semantics — both surface + // embedding failure as a warning on an otherwise-successful write. + return { + content: [ + { + type: "text" as const, + text: `Captured synthesis #${thoughtId} from ${sourceIds.length} source thoughts (embedding update failed — searchable text still saved; call capture_synthesis again if the problem persists, contact your admin, or check the Open Brain embedding service: ${embError.message})`, + }, + ], + }; + } + + return { + content: [ + { + type: "text" as const, + text: `Captured synthesis #${thoughtId} from ${sourceIds.length} source thoughts. Future queries on this topic can reuse it directly.`, + }, + ], + }; + } catch (err: unknown) { + return { + content: [ + { + type: "text" as const, + text: `Error: ${(err as Error).message}`, + }, + ], + isError: true, + }; + } + }, +); diff --git a/recipes/synthesis-capture/metadata.json b/recipes/synthesis-capture/metadata.json new file mode 100644 index 000000000..b4b184a54 --- /dev/null +++ b/recipes/synthesis-capture/metadata.json @@ -0,0 +1,26 @@ +{ + "name": "Synthesis Capture", + "description": "Adds a capture_synthesis MCP tool and POST /synthesis REST endpoint so an LLM can save its own synthesis of 3+ source thoughts as a new derived thought with full provenance back to the sources (the Query-as-Ingest pattern). Enforces anti-loop, primary-parent, and minimum-source-count safety rules.", + "category": "recipes", + "author": { + "name": "Alan Shurafa", + "github": "alanshurafa" + }, + "version": "1.0.0", + "requires": { + "open_brain": true, + "services": [], + "tools": ["Deno (Supabase Edge Functions)"] + }, + "tags": [ + "synthesis", + "provenance", + "mcp", + "rest-api", + "compound-knowledge" + ], + "difficulty": "intermediate", + "estimated_time": "30 minutes", + "created": "2026-04-17", + "updated": "2026-04-17" +} diff --git a/recipes/synthesis-capture/rest-endpoint.ts b/recipes/synthesis-capture/rest-endpoint.ts new file mode 100644 index 000000000..ece58b43a --- /dev/null +++ b/recipes/synthesis-capture/rest-endpoint.ts @@ -0,0 +1,394 @@ +/** + * POST /synthesis — REST endpoint for Open Brain + * + * Paste this into your open-brain-rest Edge Function. The exported + * `handleCaptureSynthesis` is framework-agnostic — it takes a standard + * `Request` and returns a `Response`. Wire it up to whichever routing + * style your REST function uses. + * + * For a `fetch`-style Deno.serve router, add: + * + * if (path === "/synthesis" && req.method === "POST") { + * return await handleCaptureSynthesis(req); + * } + * + * For a Hono-style router: + * + * app.post("/synthesis", (c) => handleCaptureSynthesis(c.req.raw)); + * + * What it does + * ------------ + * Accepts a pre-computed synthesis string plus 3+ source thought IDs and + * stores the synthesis as a new `public.thoughts` row with + * `derivation_layer='derived'`, `derivation_method='synthesis'`, and + * `derived_from=[...sourceIds]`. + * + * It does NOT call an LLM — synthesis generation happens client-side. This + * endpoint only persists the result with provenance. That keeps the hot path + * cheap and deterministic, and keeps secret prompts off the server. + * + * Requirements + * ------------ + * Provenance columns on `public.thoughts` (from the `provenance-chains` + * recipe schema): + * - derivation_layer (text: 'primary' | 'derived') + * - derivation_method (text: 'synthesis') + * - derived_from (jsonb: array of parent thought IDs) + * + * Assumes in scope + * ---------------- + * - `supabase` : Supabase service-role client + * - `getEmbedding(text)` : your embedding helper + * - `extractMetadata(text)` : your metadata helper + * + * Safety rules + * ------------ + * 1. `content` required (non-empty string). + * 2. `source_thought_ids` must contain at least 3 distinct positive ints. + * 3. Every source ID must exist in `public.thoughts`. + * 4. No source may have `source_type = 'synthesis'` (anti-loop). + * 5. At least one source must have `derivation_layer = 'primary'`. + */ + +type SynthesisRequestBody = { + content?: unknown; + source_thought_ids?: unknown; + question?: unknown; + topics?: unknown; + tags?: unknown; + metadata?: unknown; +}; + +function jsonResponse(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "Content-Type": "application/json" }, + }); +} + +export async function handleCaptureSynthesis(req: Request): Promise { + let body: SynthesisRequestBody; + try { + body = (await req.json()) as SynthesisRequestBody; + } catch { + return jsonResponse({ error: "Invalid JSON in request body" }, 400); + } + + const content = typeof body.content === "string" ? body.content.trim() : ""; + if (!content) { + return jsonResponse({ error: "content is required" }, 400); + } + // Size cap: 50KB matches the MCP Zod schema. Guards against accidental or + // adversarial floods that would balloon embedding and DB write costs. + // Adjust both sides (MCP + REST) together if you need longer syntheses. + if (content.length > 50_000) { + return jsonResponse( + { error: "content exceeds 50000 character limit" }, + 413, + ); + } + + // Accept either numeric BIGINT IDs or string UUIDs — OB1 installs vary. + // Stock schema in docs/01-getting-started.md uses UUID; enhanced / provenance + // variants may use BIGINT. We validate shape against whichever we see first. + const rawIds = Array.isArray(body.source_thought_ids) + ? body.source_thought_ids + : []; + // Cap raw input length BEFORE normalization so a caller cannot flood the + // normalize loop with 100k items. 50 matches the MCP schema upper bound. + if (rawIds.length > 50) { + return jsonResponse( + { error: "source_thought_ids exceeds 50 item limit" }, + 413, + ); + } + const normalized = rawIds + .map((v) => { + if (typeof v === "number" && Number.isInteger(v) && v > 0) return v; + if (typeof v === "string" && v.trim() !== "") return v.trim(); + return null; + }) + .filter((v): v is number | string => v !== null); + const sourceIds = Array.from(new Set(normalized)); + if (sourceIds.length < 3) { + return jsonResponse( + { error: "source_thought_ids must include at least 3 distinct IDs (positive integers or non-empty strings)" }, + 400, + ); + } + + // ── Safety check 1: all sources must exist ─────────────────────────────── + const { data: parents, error: parentsError } = await supabase + .from("thoughts") + .select("id, derivation_layer, source_type") + .in("id", sourceIds); + + if (parentsError) { + return jsonResponse( + { error: `parent lookup failed: ${parentsError.message}` }, + 500, + ); + } + + const parentRows = (parents ?? []) as Array<{ + id: number | string; + derivation_layer: string | null; + source_type: string | null; + }>; + const foundIds = new Set(parentRows.map((p) => p.id)); + const missing = sourceIds.filter((id) => !foundIds.has(id)); + if (missing.length > 0) { + return jsonResponse( + { error: `parent thoughts not found: ${missing.join(", ")}` }, + 400, + ); + } + + // ── Safety check 2: reject synthesis-of-synthesis ──────────────────────── + const synthSources = parentRows.filter((p) => p.source_type === "synthesis"); + if (synthSources.length > 0) { + return jsonResponse( + { + error: + "source thoughts are themselves syntheses; synthesis-of-synthesis is forbidden", + synthesis_source_ids: synthSources.map((p) => p.id), + }, + 400, + ); + } + + // ── Safety check 3: at least one primary parent ────────────────────────── + const anyPrimary = parentRows.some((p) => p.derivation_layer === "primary"); + if (!anyPrimary) { + return jsonResponse( + { + error: + "all source thoughts are derived; at least one must be primary (atomic) to prevent self-referential synthesis loops", + }, + 400, + ); + } + + // ── Input caps on optional fields (parity with README + MCP) ───────────── + // Each cap returns 413 Payload Too Large with a field-specific error so + // callers can fix the offending field without guessing. Caps match the MCP + // Zod schema so both surfaces reject identical payloads — adjust both + // handlers together if you need higher bounds. + let questionValue: string | null = null; + if (body.question !== undefined && body.question !== null) { + if (typeof body.question !== "string") { + return jsonResponse({ error: "question must be a string" }, 400); + } + const trimmedQuestion = body.question.trim(); + if (trimmedQuestion.length > 2_000) { + return jsonResponse( + { error: "question exceeds 2000 character limit" }, + 413, + ); + } + if (trimmedQuestion !== "") questionValue = trimmedQuestion; + } + + let topicsValue: string[] | null = null; + if (body.topics !== undefined && body.topics !== null) { + if (!Array.isArray(body.topics)) { + return jsonResponse({ error: "topics must be an array" }, 400); + } + if (body.topics.length > 20) { + return jsonResponse( + { error: "topics exceeds 20 item limit" }, + 413, + ); + } + for (const t of body.topics) { + if (typeof t !== "string") { + return jsonResponse( + { error: "topics entries must be strings" }, + 400, + ); + } + if (t.length > 100) { + return jsonResponse( + { error: "topics entry exceeds 100 character limit" }, + 413, + ); + } + } + topicsValue = body.topics as string[]; + } + + let tagsValue: string[] | null = null; + if (body.tags !== undefined && body.tags !== null) { + if (!Array.isArray(body.tags)) { + return jsonResponse({ error: "tags must be an array" }, 400); + } + if (body.tags.length > 20) { + return jsonResponse( + { error: "tags exceeds 20 item limit" }, + 413, + ); + } + for (const t of body.tags) { + if (typeof t !== "string") { + return jsonResponse( + { error: "tags entries must be strings" }, + 400, + ); + } + if (t.length > 50) { + return jsonResponse( + { error: "tags entry exceeds 50 character limit" }, + 413, + ); + } + } + tagsValue = body.tags as string[]; + } + + let callerMetadata: Record | null = null; + if (body.metadata !== undefined && body.metadata !== null) { + if ( + typeof body.metadata !== "object" || + Array.isArray(body.metadata) + ) { + return jsonResponse( + { error: "metadata must be a plain object" }, + 400, + ); + } + const keyCount = Object.keys(body.metadata as Record).length; + if (keyCount > 50) { + return jsonResponse( + { error: "metadata exceeds 50 key limit" }, + 413, + ); + } + callerMetadata = body.metadata as Record; + } + + // ── Build metadata, embedding, and persist ─────────────────────────────── + let embedding: number[]; + let autoMetadata: Record; + try { + [embedding, autoMetadata] = await Promise.all([ + getEmbedding(content), + extractMetadata(content), + ]); + } catch (err) { + return jsonResponse( + { error: `enrichment failed: ${(err as Error).message}` }, + 500, + ); + } + + // Build metadata in a specific order so the caller's `body.metadata` cannot + // stomp reserved provenance keys. Order: + // 1. autoMetadata (heuristic enrichment from extractMetadata) + // 2. caller-supplied body.metadata (may overlay topics/tags/notes etc.) + // 3. handler-controlled fields (question/topics/tags) — override caller + // 4. reserved provenance fields — LAST, so nothing can spoof them + const mergedMetadata: Record = { + ...autoMetadata, + }; + if (callerMetadata) { + Object.assign(mergedMetadata, callerMetadata); + } + if (questionValue) { + mergedMetadata.question = questionValue; + } + if (topicsValue) { + mergedMetadata.topics = topicsValue; + } + if (tagsValue) { + mergedMetadata.tags = tagsValue; + } + // Reserved keys — stamped LAST so body.metadata cannot overwrite them. + // These identify the write channel and provenance layer for downstream + // filtering/reporting. A caller who sets source: "capture_thought" in + // body.metadata would otherwise impersonate an MCP-atomic write. + mergedMetadata.source = "rest_synthesis"; + // Belt-and-suspenders: mirror provenance fields into metadata so they + // survive the stock `upsert_thought` RPC, which only persists + // `p_payload.metadata` and silently drops top-level `p_payload` keys. + // TODO(synthesis-capture): once the sibling `provenance-chains` recipe + // lands on main with an updated RPC that reads top-level provenance + // fields, this mirror becomes redundant and can be removed. Until then, + // this is the ONLY code path that guarantees provenance lands somewhere + // queryable on stock installs. See DEPENDENCIES.md for full rationale. + mergedMetadata.provenance = { + source_type: "synthesis", + derivation_layer: "derived", + derivation_method: "synthesis", + derived_from: sourceIds, + }; + + // Final size guard on the fully-merged metadata object. We check AFTER + // merge (including provenance stamping) so callers cannot bypass the cap + // by shrinking individual fields while still pushing the aggregate past + // 10KB. 10KB comfortably holds tens of topics/tags plus provenance while + // keeping the jsonb payload small enough that the RPC write stays cheap. + const mergedSize = JSON.stringify(mergedMetadata).length; + if (mergedSize > 10_240) { + return jsonResponse( + { error: `metadata exceeds 10KB limit (got ${mergedSize} bytes after merge)` }, + 413, + ); + } + + const { data: upsertResult, error: upsertError } = await supabase.rpc( + "upsert_thought", + { + p_content: content, + p_payload: { + source_type: "synthesis", + metadata: mergedMetadata, + derivation_layer: "derived", + derivation_method: "synthesis", + derived_from: sourceIds, + }, + }, + ); + + if (upsertError) { + return jsonResponse( + { error: `synthesis capture failed: ${upsertError.message}` }, + 500, + ); + } + + const thoughtId = (upsertResult as { id?: number | string } | null)?.id; + if (!thoughtId) { + return jsonResponse( + { error: "upsert_thought returned no thought ID" }, + 500, + ); + } + + const { error: embError } = await supabase + .from("thoughts") + .update({ embedding }) + .eq("id", thoughtId); + + if (embError) { + // Thought is safely written; the embedding is a soft-fail. Surface it so + // the caller can retry the embedding update, but keep the 200 success. + return jsonResponse( + { + thought_id: thoughtId, + source_count: sourceIds.length, + embedding_error: embError.message, + message: `Captured synthesis #${thoughtId} but embedding update failed`, + }, + 200, + ); + } + + return jsonResponse( + { + thought_id: thoughtId, + source_count: sourceIds.length, + message: `Captured synthesis #${thoughtId} from ${sourceIds.length} source thoughts`, + }, + 201, + ); +}