[Feature] Multi-model "Duet" mode: moderated Claude ↔ GPT collaboration for prompt optimisation and cross-review

[michaelbeijer]

What problem would this solve?

[Feature] Multi-model "Duet" mode: moderated Claude ↔ GPT collaboration for prompt optimisation and cross-review

Summary

Add a moderated back-and-forth mode in which two LLMs (Claude via Anthropic API + a GPT model via OpenAI API) jointly analyse and improve an artefact — primarily a project translation prompt — over a bounded number of rounds, producing (a) a full labelled transcript and (b) a final agreed deliverable with an explicit "Unresolved" section for the human to decide.

Origin: a real BRANTS patent job (WFM-001-BE-EP, 2026-07-23) where manually bouncing an AutoPrompt-generated prompt between Claude and ChatGPT produced material improvements neither model produced alone — including one outright terminology error caught (luminantiecoëfficiënt → luminance factor, not "luminance coefficient", which names a different CIE quantity), several verified source-defect catches folded into a deterministic defect table, and rejection of several "improvements" that would have broken TM-anchor fidelity. A standalone prototype exists: duet.py (working CLI relay with the debate protocol below).

Why in Workbench rather than standalone

The plumbing half already exists: the Supervertaler MCP Server's get_prompt_context and save_prompt are exactly the two endpoints a duet needs — fetch project languages, source sample, TM pairs and current prompt; save the optimised result back to the prompt library as a new version (never overwriting; the v1→v2→v3 lineage is the desired output shape). The debate loop slots between them.

Three tiers (build in this order)

Tier 1 — Prompt Duet (this issue's core scope). One run per project, pre-translation, on a ~25k-char artefact. High value, negligible cost relative to the job.

• UI: a "Duet review" action on a prompt-library entry (and/or a new MCP tool duet_optimize_prompt so it can be triggered from a Claude chat like save_prompt — possibly the cheapest v1: no new UI at all).
• Flow: get_prompt_context → N debate rounds → synthesis → transcript stored with the project → save_prompt (new version).

Tier 2 — Cross-model revision pass (separate follow-up issue). Not debate: the TEP pattern. Model A translates the batch under the locked prompt; model B reviews each target segment against source + prompt and emits pass or a flag in a distinct out-of-band marker (e.g. ⟦XR: …⟧, kept lexically separate from translator ⟦TC: …⟧ markers so QA can filter by origin). ~2× cost per batch.

Tier 3 — Per-segment arbitration (follow-up). Right-click "arbitrate this segment": full duet on a single contested segment, transcript shown. Never automated wholesale.

Debate protocol (the part that makes it work)

Two frontier models left to converse politely converge on agreement too fast (mutual-compliment collapse). The prototype counters this with a protocol enforced via system prompt + parseable verdict lines:

• Each turn must verify the other model's claims against the attached material with quoted evidence before accepting them.
• Each turn maintains a numbered OPEN ISSUES register; issues are closed only by stating a resolution.
• Each turn ends with exactly one line: VERDICT: CONTINUE or VERDICT: AGREED; AGREED is only permitted with an empty register.
• Consensus = both agents AGREED back-to-back → synthesis turn producing the deliverable between ===FINAL DELIVERABLE=== markers, with an "Unresolved (for the user to decide)" tail.
• Hard stop at N rounds (default 4); synthesis is then forced and disagreements are listed rather than resolved.

Design constraints

• Provider-agnostic engine, one module. Two thin adapters (Anthropic messages.create, OpenAI chat.completions.create) as in the prototype, so the Trados plugin and Workbench share it like the prompt library. If the engine lives in the MCP server (Python), the Trados side just calls it.
• Transcripts are project artefacts. Store under the project so a delivered translation carries its decision trail; write incrementally so a crash loses nothing.
• Cost estimate before run. The full transcript is re-sent every turn → cost grows ~quadratically with rounds. Show payload size and a rough estimate before starting; document "attach an extract, not the full document".
• Credentials. Requires OpenAI credentials alongside Anthropic in AI Settings (check whether Settings already holds both).
• UTF-8 throughout (⟦ ⟧ markers must survive console + file IO on Windows).
• Never overwrite a prompt — always save as a new library version with the duet transcript referenced.

Acceptance criteria (Tier 1)

• duet_optimize_prompt MCP tool (or Workbench action) runs a duet over the current project's prompt using get_prompt_context
• Configurable: both model strings, max rounds, max output tokens per turn, which model opens / synthesises
• Verdict-line parsing terminates on consensus or round limit
• Transcript saved incrementally under the project
• Deliverable extracted and saved via save_prompt as a new version; Unresolved section preserved
• Pre-run payload size / cost warning
• Graceful API-error retry with backoff (prototype: 3 tries, 15s·attempt)

Reference

• Prototype: duet.py (standalone CLI, working; adapter classes + protocol prompts are lift-and-shift)
• Known failure modes to test for: sycophantic early convergence (mitigation: "each model must find ≥3 substantive problems before agreement is valid" in the task brief); one model rewriting validated TM anchors during "optimisation" (the debate protocol caught this in the BRANTS session — keep it as a regression test case)

Proposed solution

Alternatives considered

No response

Additional context

No response