feat: structured-output-prefer routing strategy

CHANGELOG.md

@@ -16,7 +16,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 - `provider-exclusion` routing strategy: filters providers listed in `metadata.excluded_providers` and models listed in `metadata.excluded_models` (each a comma-separated string or list) out of the healthy domain-eligible pool before quality-first selection, emergency-retaining the highest-quality domain-eligible model when every candidate is excluded, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by Portkey / Helicone provider allow/deny lists. See `docs/guides/PROVIDER_EXCLUSION_GUIDE.md`.
 
-- `structured-output-prefer` routing strategy: when `metadata.structured_output` or `metadata.json_mode` is truthy, prefers healthy domain-eligible candidates whose capability set includes `json` (from a per-request `metadata.model_capabilities` override, the built-in known-model map, or a `gpt-5` / `claude` / `gemini` / `kimi` name heuristic), ranking by JSON support then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by LiteLLM / OpenRouter structured-output routing. See `docs/guides/STRUCTURED_OUTPUT_PREFER_GUIDE.md`.
+- `structured-output-prefer` routing strategy: when `metadata.requires_json` or `metadata.structured_output` is truthy, prefers healthy domain-eligible candidates whose capability set includes `json` / `structured` / `json_mode` (from `metadata.structured_models`, a per-request `metadata.model_capabilities` override, the built-in known-model map, or a `gpt-5` / `claude` / `gemini` / `kimi` name heuristic), ranking by structured support then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by LiteLLM / OpenRouter / Portkey structured-output routing. See `docs/guides/STRUCTURED_OUTPUT_PREFER_GUIDE.md`.
 
 - `tenant-soft-isolation` routing strategy: reads the tenant's reported request rate directly from `metadata.tenant_rpm` (or `metadata.tenant_request_rate`) and demotes tenants above `NEXUS_TENANT_SOFT_ISOLATION_RPM` (default `60`) to the lowest-cost healthy domain-compatible model instead of frontier routing — without ever rejecting the request — for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Fair-use isolation inspired by multi-tenant LLM gateways (Portkey/Helicone). See `docs/guides/TENANT_SOFT_ISOLATION_GUIDE.md`.
 

CONFIGURATION.md

@@ -2185,21 +2185,22 @@ See
 
 The `structured-output-prefer` strategy biases selection toward JSON /
 structured-output capable models when a request declares
-`metadata.structured_output` or `metadata.json_mode`.
+`metadata.requires_json` or `metadata.structured_output`.
 
 ```dotenv
 NEXUS_DEFAULT_STRATEGY=structured-output-prefer
 ```
 
 No additional `NEXUS_*` environment variables are required. Truthy metadata
 values are `true` / `1` / `yes` / `on`, or any other non-empty non-falsy
-token. Capability sets come from `metadata.model_capabilities` or the
-built-in known-model map; when neither is present for a model, names
-containing `gpt-5`, `claude`, `gemini`, or `kimi` are treated as
-JSON-capable. Ranking is `(has_json, quality, cost)`. Requests that omit
-the structured-output signal stay quality-first for GPT-5.5 /
-Claude Sonnet 4.6 / Gemini 3.x / Kimi K2. Inspired by LiteLLM / OpenRouter
-structured-output routing. See
+token. Capability sets come from `metadata.structured_models`,
+`metadata.model_capabilities`, or the built-in known-model map
+(`json`, `structured`, or `json_mode`); when neither is present for a
+model, names containing `gpt-5`, `claude`, `gemini`, or `kimi` are treated
+as structured-capable. Ranking is `(supports_structured, quality, cost)`.
+Requests that omit the structured-output signal stay quality-first for
+GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2. Inspired by LiteLLM /
+OpenRouter / Portkey structured-output routing. See
 [docs/guides/STRUCTURED_OUTPUT_PREFER_GUIDE.md](docs/guides/STRUCTURED_OUTPUT_PREFER_GUIDE.md).
 
 ## Provider-Exclusion Routing

README.md

@@ -1,6 +1,6 @@
 # nexus-llm-router
 
-![Tests](https://img.shields.io/badge/tests-762%20passing-brightgreen) ![Python](https://img.shields.io/badge/python-3.11%2B-blue) ![CI](https://github.com/Francis1998/nexus-llm-router/actions/workflows/ci.yml/badge.svg)
+![Tests](https://img.shields.io/badge/tests-765%20passing-brightgreen) ![Python](https://img.shields.io/badge/python-3.11%2B-blue) ![CI](https://github.com/Francis1998/nexus-llm-router/actions/workflows/ci.yml/badge.svg)
 
 
 > Intelligent multi-LLM routing middleware with task-aware model selection, cost optimization, fallback safety, and a drop-in OpenAI-compatible API.
@@ -198,7 +198,7 @@ Select a strategy with `X-Router-Strategy`:
 - `provider-warmup-weight`: blends quality with a per-request `metadata.provider_warmup` score (defaulting to `0.5` when omitted) using `NEXUS_PROVIDER_WARMUP_BLEND` for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
 - `tenant-soft-isolation`: tracks each tenant's rolling request rate and demotes tenants above `NEXUS_TENANT_SOFT_ISOLATION_RPM` to the cheapest healthy domain-compatible model instead of frontier routing, without rejecting the request, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
 - `model-capability-gate`: filters candidates to those whose capability set covers every capability in `metadata.required_capabilities` (for example `vision`, `tools`, `long_context`), using a per-request `metadata.model_capabilities` override or the built-in known-model map, and emergency-retains the highest-quality healthy candidate when none match for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
-- `structured-output-prefer`: when `metadata.structured_output` or `metadata.json_mode` is truthy, prefers candidates advertising a `json` capability (from `metadata.model_capabilities`, the known-model map, or a `gpt-5`/`claude`/`gemini`/`kimi` name heuristic), ranking by JSON support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
+- `structured-output-prefer`: when `metadata.requires_json` or `metadata.structured_output` is truthy, prefers candidates advertising json/structured capability (from `metadata.structured_models`, `metadata.model_capabilities` / the known-model map with `json`/`structured`/`json_mode`, or a `gpt-5`/`claude`/`gemini`/`kimi` name heuristic), ranking by structured support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
 - `provider-exclusion`: filters `metadata.excluded_providers` and `metadata.excluded_models` (comma-separated string or list) out of the healthy domain-eligible pool, then quality-first selects; emergency-retains the highest-quality eligible model when every candidate is excluded for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
 - `prompt-injection-risk-shed`: reads `metadata.prompt_injection_risk` and demotes requests at or above `NEXUS_PROMPT_INJECTION_RISK_THRESHOLD` to the cheapest healthy domain-compatible model instead of frontier routing, without rejecting the request, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
 - `thinking-model-prefer`: when `signals.complexity_score` or `metadata.complexity_score` is at or above `NEXUS_THINKING_COMPLEXITY_THRESHOLD`, prefers thinking/reasoning models (from `metadata.thinking_models` or names containing `o1`/`o3`/`reasoning`/`thinking`/`sonnet`/`opus`), ranking by thinking membership then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2

docs/guides/STRUCTURED_OUTPUT_PREFER_GUIDE.md

@@ -2,31 +2,35 @@
 
 Use `structured-output-prefer` to bias selection toward models that advertise
 JSON / structured-output support when a request declares
-`metadata.structured_output` or `metadata.json_mode`, for GPT-5.5 /
+`metadata.requires_json` or `metadata.structured_output`, for GPT-5.5 /
 Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
 
 ![Structured output prefer demo](../../assets/structured-output-prefer.gif)
 
 ## When to use it
 
-- Gateways that mirror LiteLLM / OpenRouter structured-output routing and
-  want JSON-capable models preferred for schema-constrained responses.
-- Workloads that set `json_mode` or `structured_output` upstream and still
+- Gateways that mirror LiteLLM / OpenRouter / Portkey structured-output
+  routing and want JSON-capable models preferred for schema-constrained
+  responses.
+- Workloads that set `requires_json` / `structured_output` upstream and still
   want quality-first routing when those flags are absent.
-- Fleets that maintain a per-request `metadata.model_capabilities` override
-  for models whose JSON support is not yet reflected in the built-in map.
+- Fleets that maintain a per-request `metadata.structured_models` allowlist
+  or `metadata.model_capabilities` override for models whose JSON support is
+  not yet reflected in the built-in map.
 
 ## How it works
 
-1. Read `metadata.structured_output` or `metadata.json_mode`. Truthy values
-   are `true` / `1` / `yes` / `on`, or any other non-empty non-falsy token.
+1. Read `metadata.requires_json` or `metadata.structured_output`. Truthy
+   values are `true` / `1` / `yes` / `on`, or any other non-empty non-falsy
+   token.
 2. Filter domain-eligible candidates through provider circuit health
    (emergency-retain the full eligible pool when every circuit is open).
-3. When structured output is requested, resolve each candidate's capability
-   set from `metadata.model_capabilities` or the built-in known-model map.
+3. When structured output is requested, resolve capability from
+   `metadata.structured_models`, then `metadata.model_capabilities` /
+   the built-in known-model map (`json`, `structured`, or `json_mode`).
    When neither is present for a model, treat names containing `gpt-5`,
-   `claude`, `gemini`, or `kimi` as JSON-capable.
-4. Rank by `(has_json desc, quality desc, cost asc)`.
+   `claude`, `gemini`, or `kimi` as structured-capable.
+4. Rank by `(supports_structured desc, quality desc, cost asc)`.
 5. When the structured-output signal is absent, route quality-first among
    healthy domain-eligible candidates.
 
@@ -45,7 +49,8 @@ X-Router-Strategy: structured-output-prefer
 ```json
 {
   "metadata": {
-    "json_mode": true,
+    "requires_json": true,
+    "structured_models": ["kimi-k2"],
     "model_capabilities": {
       "kimi-k2": "json"
     }

src/router/strategies.py

@@ -10436,13 +10436,13 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio
 
 
 _KNOWN_MODEL_CAPABILITIES: dict[str, frozenset[str]] = {
-    OPENAI_FRONTIER_MODEL: frozenset({"vision", "tools", "long_context"}),
+    OPENAI_FRONTIER_MODEL: frozenset({"vision", "tools", "long_context", "json"}),
     OPENAI_BALANCED_MODEL: frozenset({"tools"}),
-    ANTHROPIC_SAFETY_MODEL: frozenset({"vision", "tools", "long_context"}),
+    ANTHROPIC_SAFETY_MODEL: frozenset({"vision", "tools", "long_context", "json"}),
     ANTHROPIC_FAST_MODEL: frozenset({"tools"}),
-    GEMINI_PRO_MODEL: frozenset({"vision", "tools", "long_context"}),
-    GEMINI_FLASH_MODEL: frozenset({"vision", "tools"}),
-    MOONSHOT_BALANCED_MODEL: frozenset({"tools", "long_context"}),
+    GEMINI_PRO_MODEL: frozenset({"vision", "tools", "long_context", "json"}),
+    GEMINI_FLASH_MODEL: frozenset({"vision", "tools", "json"}),
+    MOONSHOT_BALANCED_MODEL: frozenset({"tools", "long_context", "json"}),
 }
 
 
@@ -10847,23 +10847,25 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio
 class StructuredOutputPreferStrategy(RoutingStrategy):
     """Prefer models with JSON / structured-output capability when requested.
 
-    When ``metadata.structured_output`` or ``metadata.json_mode`` is truthy
-    (``true`` / ``1`` / ``yes`` / ``on``, or any other non-empty value), rank
-    healthy domain-eligible candidates by whether they advertise a ``json``
-    capability, then by quality (descending) and cost (ascending). Capability
-    sets come from a per-request ``metadata.model_capabilities`` override or
-    the built-in known-model map when present; otherwise models whose names
-    contain ``gpt-5``, ``claude``, ``gemini``, or ``kimi`` are treated as
-    JSON-capable. Requests that omit the structured-output signal stay
-    quality-first. Inspired by LiteLLM / OpenRouter structured-output routing
-    for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
+    When ``metadata.requires_json`` or ``metadata.structured_output`` is
+    truthy (``true`` / ``1`` / ``yes`` / ``on``, or any other non-empty
+    non-falsy token), rank healthy domain-eligible candidates by whether they
+    support structured/JSON output, then by quality (descending) and cost
+    (ascending). Capability is resolved from ``metadata.structured_models``,
+    ``metadata.model_capabilities`` / the built-in known-model map
+    (``json``, ``structured``, or ``json_mode``), or a name heuristic matching
+    ``gpt-5``, ``claude``, ``gemini``, or ``kimi``. Requests that omit the
+    structured-output signal stay quality-first. Inspired by LiteLLM /
+    OpenRouter / Portkey structured-output routing for GPT-5.5 /
+    Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
     """
 
     strategy_name = RoutingStrategyName.STRUCTURED_OUTPUT_PREFER
 
     _TRUTHY_TOKENS = frozenset({"true", "1", "yes", "on"})
     _FALSY_TOKENS = frozenset({"false", "0", "no", "off", ""})
     _JSON_NAME_TOKENS = ("gpt-5", "claude", "gemini", "kimi")
+    _STRUCTURED_CAPABILITIES = frozenset({"json", "structured", "json_mode"})
 
     def __init__(
         self,
@@ -10878,27 +10880,42 @@ def __init__(
             _KNOWN_MODEL_CAPABILITIES if capability_map is None else capability_map
         )
 
+    @classmethod
+    def _is_truthy(cls, value: object) -> bool:
+        """Return whether a metadata value is treated as truthy."""
+        if value is None:
+            return False
+        if isinstance(value, bool):
+            return value
+        if isinstance(value, (int, float)) and not isinstance(value, bool):
+            return value != 0
+        if isinstance(value, (list, tuple, set, dict)):
+            return len(value) > 0
+        text = str(value).strip().lower()
+        if text in cls._FALSY_TOKENS:
+            return False
+        return text in cls._TRUTHY_TOKENS or bool(text)
+
     @classmethod
     def _wants_structured_output(cls, request: RouterRequest) -> bool:
         """Return whether the request asks for structured / JSON output."""
-        for key in ("structured_output", "json_mode"):
-            value = request.metadata.get(key)
-            if value is None:
-                continue
-            if isinstance(value, bool):
-                if value:
-                    return True
-                continue
-            if isinstance(value, (int, float)) and not isinstance(value, bool):
-                if value != 0:
-                    return True
-                continue
-            text = str(value).strip().lower()
-            if text in cls._FALSY_TOKENS:
-                continue
-            if text in cls._TRUTHY_TOKENS or text:
-                return True
-        return False
+        return cls._is_truthy(request.metadata.get("requires_json")) or cls._is_truthy(
+            request.metadata.get("structured_output")
+        )
+
+    @staticmethod
+    def _structured_allowlist(request: RouterRequest) -> frozenset[str] | None:
+        """Parse an optional ``metadata.structured_models`` allowlist."""
+        raw = request.metadata.get("structured_models")
+        if raw is None:
+            return None
+        if isinstance(raw, str):
+            parts: Iterable[object] = raw.split(",")
+        elif isinstance(raw, Iterable) and not isinstance(raw, (bytes, bytearray)):
+            parts = raw
+        else:
+            return frozenset()
+        return frozenset(stripped.lower() for item in parts if (stripped := str(item).strip()))
 
     def _capabilities_for(self, model: str, request: RouterRequest) -> frozenset[str] | None:
         """Resolve an explicit capability set, or ``None`` when absent."""
@@ -10916,16 +10933,19 @@ def _capabilities_for(self, model: str, request: RouterRequest) -> frozenset[str
             return self._capability_map[model]
         return None
 
-    def _has_json(self, model: str, request: RouterRequest) -> bool:
+    def _supports_structured(self, model: str, request: RouterRequest) -> bool:
         """Return whether a model is treated as JSON / structured-output capable."""
+        allowlist = self._structured_allowlist(request)
+        if allowlist is not None:
+            return model.lower() in allowlist
         capabilities = self._capabilities_for(model, request)
         if capabilities is not None:
-            return "json" in capabilities
+            return bool(capabilities & self._STRUCTURED_CAPABILITIES)
         lower = model.lower()
         return any(token in lower for token in self._JSON_NAME_TOKENS)
 
     def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecision:
-        """Prefer JSON-capable models when structured output is requested."""
+        """Prefer structured-output models when JSON / schema output is requested."""
         wants_structured = self._wants_structured_output(request)
         eligible = [
             candidate
@@ -10945,16 +10965,17 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio
             )
             for candidate in active
         }
-        json_flags = {
-            candidate.model: self._has_json(candidate.model, request) for candidate in active
+        structured_flags = {
+            candidate.model: self._supports_structured(candidate.model, request)
+            for candidate in active
         }
         availability_note = "healthy" if healthy else "circuit-open emergency"
 
         if wants_structured:
             selected = max(
                 active,
                 key=lambda candidate: (
-                    json_flags[candidate.model],
+                    structured_flags[candidate.model],
                     candidate.quality_score,
                     -costs[candidate.model],
                     candidate.model,
@@ -10963,13 +10984,17 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio
             fallback_candidates = sorted(
                 (candidate for candidate in active if candidate.model != selected.model),
                 key=lambda candidate: (
-                    not json_flags[candidate.model],
+                    not structured_flags[candidate.model],
                     -candidate.quality_score,
                     costs[candidate.model],
                     candidate.model,
                 ),
             )
-            json_note = "json-capable" if json_flags[selected.model] else "non-json fallback"
+            json_note = (
+                "structured-capable"
+                if structured_flags[selected.model]
+                else "non-structured fallback"
+            )
             rationale = (
                 f"structured-output-prefer requested; selected {availability_note} "
                 f"{json_note} {selected.model} (quality {selected.quality_score:.2f})"
@@ -10992,7 +11017,7 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio
                 ),
             )
             rationale = (
-                f"structured-output-prefer no structured_output/json_mode signal; "
+                f"structured-output-prefer no requires_json/structured_output signal; "
                 f"selected {availability_note} quality-first {selected.model}"
             )