synth: a data synthesizer — training data for traffic that doesn't exist yet - #14
Open
pradipta-lyzr wants to merge 14 commits into
Open
synth: a data synthesizer — training data for traffic that doesn't exist yet#14pradipta-lyzr wants to merge 14 commits into
pradipta-lyzr wants to merge 14 commits into
Conversation
… unit
Three fixes the synthesizer needs underneath it, each standing on its own.
traces._span_call hardcoded tools=None, so a Trajectory rebuilt from spans
never carried the tool definitions the spans were already reporting — and
to_dataset therefore never emitted a "tools" key. Read them from
gen_ai.request.tools, or OpenInference's indexed llm.tools.{i}.tool.json_schema,
wrapping bare function schemas.
more_plus.split_units built a unit's BM25 surrogate from grp[0] alone, so when
a unit holds several phrasings of one fact, rows 2..k contributed nothing to
routing — an expert reachable only by the wording that happened to come first.
Join the whole group.
Neither backend passes a dataset's tool schemas to the training chat template,
while chat() does pass them at inference. Whether trl can carry a tools column
is version-dependent and unverified here, so say it out loud rather than let
the skew pass silently — the convention this repo already holds for config a
backend can't honor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t yet
capture() records a live agent, traces reads one that already ran, and
Dataset.from_* loads what you already have. All three need the traffic to
exist. This makes it: describe the task, point at a document, or amplify a few
real episodes, and a teacher writes the rest.
Two orthogonal axes again, mirroring backends × methods. Seeds decide where
scenarios come from (task / document / episodes); modes decide what gets
written per scenario (conversation / preference / paraphrases). Any seed
composes with any mode. Everything converges on Trajectory — the same type
capture and traces produce — and emit.py renders that into the shape the
consumer takes, chosen from the method's *spec* and never its name.
Generation is taxonomy first, instances second. A teacher asked for variety in
one breath rewrites one example n times; a teacher asked to fill a named slot
in a scenario tree doesn't. That structure, not prompt wording, is the
anti-mode-collapse mechanism.
Two consequences of the architecture worth naming:
- trajectory-GRPO learns from the spread between good and bad attempts, so
the judge scores but does not filter for that format — gating it would
throw away exactly the signal it needs.
- an outcome is accepted or rejected whole. MoRE+ groups rows by fixed size,
so dropping one row of a paraphrase unit would misalign every unit after
it; since a conversation outcome is one row, that is the same rule
everywhere.
Nothing is dropped silently: SynthReport reconciles exactly (generated == kept
+ invalid + duplicate + low-scoring + surplus) and report.balanced asserts it.
Teachers are duck-typed .chat(), so a frontier model, a local slm.load() model
and — later, for SDPO — the student itself are interchangeable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`shadowlm synth` with --dry-run, so you can see the resolved output shape before spending a teacher call on it. POST/GET /v1/synth follows the model-download pattern — a background thread polled for status — rather than the training queue, since synthesis is teacher calls and would otherwise hold the one training slot for the whole run. The finished rows land in DatasetStore like any other dataset. Teacher API keys are used for the run and never written to disk, unlike the HF token, which is deliberately persisted. Datasets gains a Synthesize tab beside Upload and Hugging Face: task, optional grounding document, target method, and a teacher that is either an OpenAI-compatible endpoint or a model already on this machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ples synthesize_from_task.py is the cold start — no agent has run, so there is nothing to capture and no traces to read. synthesize_from_doc.py is the signature path: facts pulled from a document, each phrased several ways, because MoRE+ routes on the question side and a fact asked about one way is an expert nobody can reach. CLAUDE.md gains the synth/ section and, while there, the traces.py and eval.py entries the architecture notes had drifted past. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything below was found by running the synthesizer against a real model.
A scripted fake answers the prompt you meant to write; a real one answers the
prompt you actually wrote.
Tool episodes failed 6/6. Three causes, all mine:
- The OpenAI wire shape is four levels deep — messages > tool_calls >
function > arguments — and the teacher miscounted the closing braces,
emitting `"arguments":{"invoice_id":"5678"}}}]}}` and losing the whole
episode to a JSON error. It also invented call ids it then failed to match.
So ask for a shallow {"call": {name, args}} / {"role":"tool","result":…}
pair and build the protocol in _wire_tool_calls, where the nesting and the
ids are right by construction. The teacher writes the semantics; we write
the wire format.
- A reply cut off by the token budget left the outer object unclosed, so the
object scan returned the *first message* instead: a dict that parses,
carries no conversation, and reads as "the teacher said nothing useful".
_parse_conversation now requires the shape, turning a silent dead end into
a retry.
- The judge scored only first-user-message against last-assistant-message, so
an answer citing what a tool returned looked unsupported and was gated out.
It now sees the whole lead-up — which is also the fix for any multi-turn
episode, not just tool ones.
Offered a "tool" role with no tools defined, the teacher invented tool turns to
narrate its own reasoning ("Classifying urgency: urgent"). Don't offer the role
when there are no tools, and say why.
per_scenario now defaults by shape rather than to 4. GRPO and the MoRE methods
want depth — several attempts to compare, several phrasings to route by —
but four conversations about one scenario are four rewrites of one example.
Measured on the same workload: 4 scenarios → 7, wasted retries 5 → 1.
Also cap the turn count in the prompt: max_seq_length defaults to 2048, so a
sprawling episode would have its final assistant turn — the thing being trained
— truncated away at training time.
Tool episodes now land 6/6 with correct wire form, the OTLP round-trip is
lossless on real generated data, and document→MoRE+ produces exactly k rows per
fact with every unit routable by its keyword-style phrasing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ld ask Leading the question prompt with "TASK: <what the assistant does>" made the teacher perform the task instead of writing the customer's message. The "question" came back as an assistant reply, "chosen" then answered that reply in almost the same words, and the pair collapsed into three near-identical strings that teach a model nothing. Both sides passed the not-equal check by differing somewhere past the first two hundred characters. So state the writing job first and demote the task to parenthetical context, and reject a pair whose question and chosen answer overlap heavily — the signature of the teacher having swapped roles again. The reason this survived until a live run: no fake-teacher test exercised the preference path at all, so renaming the prompt broke nothing visible. Covered now, including the student-as-rejected pairing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ask for JSON
A pass over the feature with fresh eyes, plus a live run to check each fix.
Robustness — the expensive failure was losing a run's paid-for rows:
- One teacher exception anywhere used to propagate through the thread pool
and kill the whole run at whatever it cost. Now a failed job counts as
invalid and the run continues; a circuit breaker trips after four
consecutive failures so a dead endpoint stops the run early *with* its
rows instead of burning a retry cycle per remaining job. Generation and
judging get separate breakers — a generation failure must not cancel the
scoring of rows already generated.
- serve.start_synth loaded a local teacher (and resolved an HF episodes
dataset) on the HTTP request thread; a cold 7B load held the POST for
minutes. Both now happen on the background thread the run owns.
Accounting — two places the funnel quietly lied:
- A GRPO group whose attempts all scored the same is dropped at emit (no
spread, no signal), but report.kept still claimed its rows. New
rejected_flat bucket; report.balanced now covers it. A live run promptly
produced one: "4 flat-group", reconciled.
- mean_score skipped rows scoring 0.0 (`if t.reward`), biasing the mean up
exactly where low scores are legitimate. Now averages judge_score metrics.
Coverage and cost:
- Document seeds ignored `avoid`: a top-up round re-extracted and
re-planned the same facts, dedup rejected them all, and the run stalled
without ever reading past where round one stopped. Facts are now cached
per chunk and skipped once planned, so round two advances into the
document.
- The CLI's --per-scenario default of 4 silently overrode the shape-aware
default (breadth for SFT, depth for GRPO/MoRE). It defaults to unset now,
and --dry-run prints the resolved value.
Quality:
- Student-as-rejected pairs are now judged on both sides; a pair where the
student scores no worse than the teacher is dropped — it would train the
DPO gap backwards. Teacher-corrupted rejects stay unjudged: flawed by
construction, scoring them doubles judge cost for nothing.
- OpenAI-compatible teachers ask for the server's JSON mode on
JSON-shaped prompts, with one plain retry (remembered) for servers that
reject response_format. Live effect: tool episodes went from 1 invalid +
1 repaired to 0 and 0.
- The paraphrase prompt now actually uses the style its metadata was
already claiming; serve treats min_score=0 as "no gate", same as the CLI;
to_groups' error named a parameter that no longer exists.
Without it wc -l undercounts by one — the report says 6 rows, the file reports 5 — and anything appending to the file corrupts the final row.
serve, dev and demo run $(VENV)/bin/shadowlm; on a fresh clone that is just a missing file, and make reports it as one. Point at make install / install-torch instead, and mention that the server also runs straight from any environment that already has the package. check and gpu-test only need the interpreter, so they now bootstrap the venv through the existing rule rather than failing the same way.
The wheel ships the compiled UI, so the built bundle is part of the feature, not a local artifact.
An empty key was accepted silently and only surfaced as a raw provider 401 once the run was already underway — a message that says nothing about where the key was supposed to go. Check it when the teacher is built instead, and name the three places it can come from. Only api.openai.com insists: vLLM and Ollama serve the same API without auth, so a custom base_url stays keyless.
…ument The studio sat at 0 for an entire run. on_progress fired once per round, and round one does all the work — so the bar had nothing to show until everything was already done, then jumped to 100%. Generation was parallel, but ThreadPoolExecutor.map hands back the whole batch at once, so there was no incremental signal even internally. run_jobs now collects through as_completed while writing results into their submitted slot: callers still get submission order (MoRE+ units are consecutive rows and must stay that way) and get a tick per finished job. on_progress becomes (done, total, phase) — planning, generating, judging, kept — and fires from every phase including the leaf planning. The studio bar tracks the live phase counter while a batch runs and the real kept count once a round lands; the server keeps logging one line per round rather than one per job, which would bury the report. Document fact extraction ran one chunk at a time, so a long document spent its whole extraction latency up front with the teacher idle. It now goes through run_jobs like everything else. Default frontier parallelism 4 → 8. Measured on the handbook document: longest silent stretch 7.1s → 5.2s, and that remainder is one indivisible extraction call, now shown as a labelled pulsing phase rather than a dead bar.
Three things stood between this and production, all of them about a feature whose entire cost model is "call a paid API in a loop". You could not see what a run cost. The provider reports usage on every response and we discarded it. Tokens now come straight from that block into the report — prompt and completion split out, and deliberately no price table, because a hardcoded rate card goes stale and then lies about money. A teacher that reports no usage (a local model) reads as zero rather than a guess. You could not stop one. A 5,000-row run started by mistake had to be killed with the server. synthesize() takes should_stop= and token_budget=; the studio grows a cancel button and the CLI a --token-budget flag. Both keep the rows already produced. They gate generation only, and that distinction is the whole design: rows that exist have already been paid for, so leaving them unscored fails them at the judge gate and wastes everything spent making them. Scoring what was produced costs a fraction of generating it. The test suite pins this — a generation outage must still let judging finish. The budget is honestly a throttle rather than a ceiling. Measured live, a 1,552-token budget landed at 8,587: in-flight parallel calls plus that deliberate scoring. Said plainly in the docstring and the CLI help rather than dressed up as a cap. Studio runs were memory-only and unbounded — lost on restart, growing forever. They now persist under work_root/synth/ like training jobs, with the log lines per run and the runs themselves both capped, and a record still marked running at startup is reported stopped, since its thread is gone. Two smaller things a real provider will hand you: 429s now honour the server's own Retry-After instead of our backoff curve, and a 400 naming max_completion_tokens switches the parameter and retries, so reasoning models work without configuration.
Only the docs collided, and only where upstream and this branch each added a row describing an inlet — trace ingestion and synthesis are both real, so both stay. Everything else merged clean; the suite is green at 293 tests. Worth noting for the review: upstream independently fixed the worker's backend_factory bug the same way this branch's sibling fix did, so fix/worker-honors-backend-factory is now redundant and can be dropped. Adopted from upstream on the way through: the login throttle, owner-only settings/token files, the request body cap and the tar member vetting all now sit under the synth routes too, which read their bodies through the same capped helper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fourth inlet.
capture()records a live agent,tracesreads one thatalready ran,
Dataset.from_*loads what you have — all three need the trafficto exist. This makes it: describe the task, point at a document, or amplify a
few real episodes, and a teacher model writes the rest.
Seeds × modes, mirroring backends × methods. Everything converges on
Trajectoryandemit.pyrenders it into the shape the consumer takes, chosenfrom the method's spec, never its name — so
method="dpo"yields preferencepairs and
method="more_plus"yields query-diverse paraphrase units.format="otlp"emits OpenTelemetry GenAI spans that round-trip losslesslythrough
traces.from_otlp— verified on real generated data, tool calls and all.Reachable from the SDK,
shadowlm synth,POST /v1/synth, and a Synthesize tabin the studio. 174 tests; every emitter is asserted by the code that consumes it.
Also fixes three things underneath:
traces.pynever extracted tool schemas,MoRE+ routed on only the first phrasing in a unit, and tool-schema train/inference
skew is now stated rather than silent.