Skip to content

feat(llm-sdk): Anthropic forced-tool fallback for models without output_config #96

@Sahil5963

Description

@Sahil5963

Context

PR #95 added native output_config.format support for Anthropic structured output (Claude 3.5 / 3.7 / 4 series). This works on:

  • Anthropic Claude API ✅
  • AWS Bedrock ✅

It does NOT work on:

Problem

Customers on Vertex AI or pinned to older Claude 3 models will silently get plain text instead of structured JSON when they pass responseFormat. The capability gate warns but doesn't actually return JSON.

Proposed fix

Implement the forced-tool pattern (matches Vercel AI SDK's behavior):

  1. When responseFormat.type === "json_schema" AND model lacks native output_config support, inject a synthetic tool whose input_schema is the user's response schema:
    tools: [{ name: "json", description: "Respond as JSON.", input_schema: <user schema> }]
    tool_choice: { type: "tool", name: "json" }
  2. In response parsing, extract the tool call's input and surface it as content so the SDK behaves identically across all paths.

Reference

Vercel's implementation: packages/anthropic/src/anthropic-language-model.ts:336-352 (in their repo). They gate via structuredOutputMode: 'auto' | 'outputFormat' | 'tool'.

Files to touch

  • packages/llm-sdk/src/adapters/anthropic.ts — add fallback path in buildRequestOptions + tool extraction in complete() / stream parser
  • packages/llm-sdk/src/providers/anthropic/provider.ts — same in doGenerate / doStream
  • packages/llm-sdk/src/providers/anthropic/provider.ts — flip older Claude 3 models from jsonMode: falsetrue once fallback is wired

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions