response_format: json_object is a no-op on the default OpenRouter model
Summary
extractEntities() sends response_format: { type: "json_object" } on the OpenRouter call, with the comment that it's there because proxied models otherwise wrap JSON in prose. The default model on that path is anthropic/claude-haiku-4-5 (CLASSIFIER_MODEL_OPENROUTER in _shared/config.ts), and Anthropic models don't implement that OpenAI-shaped parameter. It passes through and is ignored.
So the guard the comment describes isn't active in the default configuration. Nothing forces JSON, and parseExtractionResult throws whenever the model returns anything else.
Reproducing
The failure isn't random, which is what makes it worth reporting. It reproduces on thought content that reads as an instruction.
I ran the worker over ~800 thoughts. Exactly one failed, and it failed all 5 attempts identically. Its content was a directive written to an LLM about how a downstream system should behave, phrased with "do not refuse" and "this supersedes" language. Roughly 750 characters, nothing exotic.
The model appears to respond to the directive rather than extracting entities from it, returns prose, and the parse fails. wrapThoughtContent() does its job here, the content isn't escaping its tags, but wrapping doesn't stop a model from treating wrapped text as something to answer.
Every other thought in the corpus extracted fine, including ones with far more unusual content. The common factor in the failure is that the text is instruction-shaped.
This class of content isn't unusual in a personal knowledge base. Notes about how you want your tooling to behave are a normal thing to write down, and every one of them is a candidate for this failure.
Possible directions
Not proposing a specific fix, since the tradeoffs are yours to weigh:
- Retry once on parse failure, appending something like "Return only the JSON object, no other text" to the prompt. Cheapest option. Costs an extra call only on the failure path.
- Use Anthropic's native tool-use / structured output on the Anthropic path rather than the OpenAI-shaped parameter. Correct fix, larger change, and it forks the request-building code per provider.
- Leave the behavior, fix the comment, so the next person doesn't assume the guard is active. Fine on its own if neither of the above is worth the complexity.
Happy to open a PR for any of these if one is preferred.
Environment
- Supabase Edge Functions
- OpenRouter as the only configured provider
anthropic/claude-haiku-4-5 (default)- ~800 thoughts, 1 deterministic failure
Related
Separate PR fixes the error reporting that made this hard to diagnose: the failure surfaced as "No LLM API key configured" rather than the real OpenRouter error.
response_format: json_objectis a no-op on the default OpenRouter modelSummary
extractEntities()sendsresponse_format: { type: "json_object" }on the OpenRouter call, with the comment that it's there because proxied models otherwise wrap JSON in prose. The default model on that path isanthropic/claude-haiku-4-5(CLASSIFIER_MODEL_OPENROUTERin_shared/config.ts), and Anthropic models don't implement that OpenAI-shaped parameter. It passes through and is ignored.So the guard the comment describes isn't active in the default configuration. Nothing forces JSON, and
parseExtractionResultthrows whenever the model returns anything else.Reproducing
The failure isn't random, which is what makes it worth reporting. It reproduces on thought content that reads as an instruction.
I ran the worker over ~800 thoughts. Exactly one failed, and it failed all 5 attempts identically. Its content was a directive written to an LLM about how a downstream system should behave, phrased with "do not refuse" and "this supersedes" language. Roughly 750 characters, nothing exotic.
The model appears to respond to the directive rather than extracting entities from it, returns prose, and the parse fails.
wrapThoughtContent()does its job here, the content isn't escaping its tags, but wrapping doesn't stop a model from treating wrapped text as something to answer.Every other thought in the corpus extracted fine, including ones with far more unusual content. The common factor in the failure is that the text is instruction-shaped.
This class of content isn't unusual in a personal knowledge base. Notes about how you want your tooling to behave are a normal thing to write down, and every one of them is a candidate for this failure.
Possible directions
Not proposing a specific fix, since the tradeoffs are yours to weigh:
Happy to open a PR for any of these if one is preferred.
Environment
anthropic/claude-haiku-4-5(default)Related
Separate PR fixes the error reporting that made this hard to diagnose: the failure surfaced as "No LLM API key configured" rather than the real OpenRouter error.