diff --git a/CHANGELOG.md b/CHANGELOG.md index 39e2bca..c3ef5a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] ### Added +- `audio-input-prefer` routing strategy: when `metadata.requires_audio`, `metadata.audio_input`, or `metadata.audio` is truthy, prefers healthy domain-eligible candidates whose capability set includes `audio` (from `metadata.audio_models`, a per-request `metadata.model_capabilities` override, the built-in known-model map, or an `audio` / `realtime` / `gpt-4o-audio` / `gemini` name heuristic), ranking by audio support then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by OpenRouter / LiteLLM / Portkey audio-input capability routing. See `docs/guides/AUDIO_INPUT_PREFER_GUIDE.md`. - `long-context-prefer` routing strategy: when `metadata.min_context_tokens` is a positive int (or `metadata.long_context` is truthy with default threshold `100000`), prefers healthy domain-eligible candidates whose context window meets the threshold (from `metadata.model_context_windows`, the built-in known-model map / catalog `context_window`, or a `gpt-5` / `claude` / `gemini` / `kimi` name heuristic defaulting to `200000`), ranking by context then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by OpenRouter / LiteLLM / Portkey long-context capability routing. See `docs/guides/LONG_CONTEXT_PREFER_GUIDE.md`. - `streaming-prefer` routing strategy: when `metadata.stream` or `metadata.requires_streaming` is truthy, prefers healthy domain-eligible candidates whose capability set includes `streaming` (from `metadata.streaming_models`, a per-request `metadata.model_capabilities` override, the built-in known-model map, or a `gpt-5` / `claude` / `gemini` / `kimi` name heuristic), ranking by streaming support then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by OpenRouter / LiteLLM / Portkey streaming capability routing. See `docs/guides/STREAMING_PREFER_GUIDE.md`. - `multimodal-input-prefer` routing strategy: when `metadata.has_images` or `metadata.has_audio` is truthy, prefers healthy domain-eligible candidates whose capability set includes `vision` / `audio` / `multimodal` (from `metadata.multimodal_models`, a per-request `metadata.model_capabilities` override, the built-in known-model map, or a `gpt-5` / `claude` / `gemini` / `vision` name heuristic), ranking by multimodal support then quality then cost; otherwise stays quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic. Inspired by OpenRouter multimodal routing. See `docs/guides/MULTIMODAL_INPUT_PREFER_GUIDE.md`. diff --git a/CONFIGURATION.md b/CONFIGURATION.md index c7fe48a..02339ce 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -2317,3 +2317,5 @@ models when a request declares `metadata.stream` or The `long-context-prefer` strategy biases selection toward models whose context window meets a requested threshold when a request declares `metadata.min_context_tokens` or `metadata.long_context`. + +The `audio-input-prefer` strategy biases selection toward audio-capable models when a request declares `metadata.requires_audio`, `metadata.audio_input`, or `metadata.audio`. diff --git a/README.md b/README.md index 2030020..c34fa50 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # nexus-llm-router -![Tests](https://img.shields.io/badge/tests-787%20passing-brightgreen) ![Python](https://img.shields.io/badge/python-3.11%2B-blue) ![CI](https://github.com/Francis1998/nexus-llm-router/actions/workflows/ci.yml/badge.svg) +![Tests](https://img.shields.io/badge/tests-799%20passing-brightgreen) ![Python](https://img.shields.io/badge/python-3.11%2B-blue) ![CI](https://github.com/Francis1998/nexus-llm-router/actions/workflows/ci.yml/badge.svg) > Intelligent multi-LLM routing middleware with task-aware model selection, cost optimization, fallback safety, and a drop-in OpenAI-compatible API. @@ -206,6 +206,7 @@ Select a strategy with `X-Router-Strategy`: - `thinking-model-prefer`: when `signals.complexity_score` or `metadata.complexity_score` is at or above `NEXUS_THINKING_COMPLEXITY_THRESHOLD`, prefers thinking/reasoning models (from `metadata.thinking_models` or names containing `o1`/`o3`/`reasoning`/`thinking`/`sonnet`/`opus`), ranking by thinking membership then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 - `tool-calling-prefer`: when `metadata.requires_tools` is truthy or `metadata.tools` is non-empty, prefers candidates advertising tool/function calling (from `metadata.tool_capable_models`, `metadata.model_capabilities` / the known-model map, or a `gpt-5`/`claude`/`gemini`/`kimi` name heuristic), ranking by tool support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 - `multimodal-input-prefer`: when `metadata.has_images` or `metadata.has_audio` is truthy, prefers candidates advertising vision/multimodal capability (from `metadata.multimodal_models`, `metadata.model_capabilities` / the known-model map, or a `gpt-5`/`claude`/`gemini`/`vision` name heuristic), ranking by multimodal support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 +- `audio-input-prefer`: when `metadata.requires_audio`, `metadata.audio_input`, or `metadata.audio` is truthy, prefers candidates advertising audio capability (from `metadata.audio_models`, `metadata.model_capabilities` / the known-model map, or an `audio`/`realtime`/`gpt-4o-audio`/`gemini` name heuristic), ranking by audio support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 - `ab`: deterministic request-id buckets across two model arms ## Documentation diff --git a/assets/audio-input-prefer.gif b/assets/audio-input-prefer.gif new file mode 100644 index 0000000..94a578c Binary files /dev/null and b/assets/audio-input-prefer.gif differ diff --git a/docs/guides/AUDIO_INPUT_PREFER_GUIDE.md b/docs/guides/AUDIO_INPUT_PREFER_GUIDE.md new file mode 100644 index 0000000..b9646c4 --- /dev/null +++ b/docs/guides/AUDIO_INPUT_PREFER_GUIDE.md @@ -0,0 +1,58 @@ +# Audio-Input-Prefer Routing Guide + +Use `audio-input-prefer` to bias selection toward `audio`-capable +models when a request declares `metadata.requires_audio` / `metadata.audio_input` / `metadata.audio`, for GPT-5.5 / Claude Sonnet 4.6 / +Gemini 3.x / Kimi K2. + +![audio-input-prefer demo](../../assets/audio-input-prefer.gif) + +## When to use it + +- Gateways that mirror OpenRouter / LiteLLM / Portkey audio-input capability routing and want + `audio`-capable models preferred when the signal is set. +- Workloads that set `requires_audio`, `audio_input`, `audio` upstream and still want quality-first + routing when those flags are absent. +- Fleets that maintain a per-request `metadata.audio_models` allowlist + or `metadata.model_capabilities` override for models whose + `audio` support is not yet reflected in the built-in map. + +## How it works + +1. Read `metadata.requires_audio` / `metadata.audio_input` / `metadata.audio`. Truthy values are `true` / `1` / `yes` / `on`, or any + other non-empty non-falsy token. +2. Filter domain-eligible candidates through provider circuit health + (emergency-retain the full eligible pool when every circuit is open). +3. When the signal is present, resolve capability from + `metadata.audio_models`, then `metadata.model_capabilities` / + the built-in known-model map (`audio`). When neither is + present for a model, treat names containing `audio`, `realtime`, `gpt-4o-audio`, `gemini` as capable. +4. Rank by `(supports_audio desc, quality desc, cost asc)`. +5. When the signal is absent, route quality-first among healthy + domain-eligible candidates. + +## Quick start + +```bash +export NEXUS_DEFAULT_STRATEGY=audio-input-prefer +``` + +Or select it per request: + +```http +X-Router-Strategy: audio-input-prefer +``` + +```json +{ + "metadata": { + "requires_audio": true, + "audio_models": ["gemini-3.5-flash"], + "model_capabilities": { + "gemini-3.5-flash": "audio" + } + } +} +``` + +No additional `NEXUS_*` environment variables are required — selection is +driven entirely by request metadata. diff --git a/src/router/schemas.py b/src/router/schemas.py index 3b80bea..5c812c3 100644 --- a/src/router/schemas.py +++ b/src/router/schemas.py @@ -121,6 +121,7 @@ class RoutingStrategyName(StrEnum): MULTIMODAL_INPUT_PREFER = "multimodal-input-prefer" STREAMING_PREFER = "streaming-prefer" LONG_CONTEXT_PREFER = "long-context-prefer" + AUDIO_INPUT_PREFER = "audio-input-prefer" AB_TEST = "ab" diff --git a/src/router/strategies.py b/src/router/strategies.py index fc86b55..18c5e2b 100644 --- a/src/router/strategies.py +++ b/src/router/strategies.py @@ -10436,12 +10436,14 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio _KNOWN_MODEL_CAPABILITIES: dict[str, frozenset[str]] = { - OPENAI_FRONTIER_MODEL: frozenset({"vision", "tools", "long_context", "json", "streaming"}), + OPENAI_FRONTIER_MODEL: frozenset( + {"vision", "tools", "long_context", "json", "streaming", "audio"} + ), OPENAI_BALANCED_MODEL: frozenset({"tools", "streaming"}), ANTHROPIC_SAFETY_MODEL: frozenset({"vision", "tools", "long_context", "json", "streaming"}), ANTHROPIC_FAST_MODEL: frozenset({"tools", "streaming"}), - GEMINI_PRO_MODEL: frozenset({"vision", "tools", "long_context", "json", "streaming"}), - GEMINI_FLASH_MODEL: frozenset({"vision", "tools", "json", "streaming"}), + GEMINI_PRO_MODEL: frozenset({"vision", "tools", "long_context", "json", "streaming", "audio"}), + GEMINI_FLASH_MODEL: frozenset({"vision", "tools", "json", "streaming", "audio"}), MOONSHOT_BALANCED_MODEL: frozenset({"tools", "long_context", "json", "streaming"}), } @@ -12164,6 +12166,192 @@ def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecisio ) +class AudioInputPreferStrategy(RoutingStrategy): + """Prefer audio-capable models when requested. + + When ``metadata.requires_audio``, ``metadata.audio_input``, or + ``metadata.audio`` is truthy, rank healthy domain-eligible candidates by + whether they support ``audio``, then by quality (descending) and cost + (ascending). Capability is resolved from ``metadata.audio_models``, + ``metadata.model_capabilities`` / the built-in known-model map + (``audio`` capability), or a name heuristic matching ``audio``, + ``realtime``, ``gpt-4o-audio``, or ``gemini``. Requests that omit the + signal stay quality-first. Inspired by OpenRouter / LiteLLM / Portkey + audio-input capability routing for GPT-5.5 / Claude Sonnet 4.6 / + Gemini 3.x / Kimi K2. + """ + + strategy_name = RoutingStrategyName.AUDIO_INPUT_PREFER + + _TRUTHY_TOKENS = frozenset({"true", "1", "yes", "on"}) + _FALSY_TOKENS = frozenset({"false", "0", "no", "off", ""}) + _NAME_TOKENS = ( + "audio", + "realtime", + "gpt-4o-audio", + "gemini", + ) + + def __init__( + self, + model_catalog: Mapping[str, ModelCandidate], + provider_health: ProviderHealth, + capability_map: Mapping[str, frozenset[str]] | None = None, + ) -> None: + """Initialize audio-input-prefer routing.""" + super().__init__(model_catalog) + self._provider_health = provider_health + self._capability_map: Mapping[str, frozenset[str]] = ( + _KNOWN_MODEL_CAPABILITIES if capability_map is None else capability_map + ) + + @classmethod + def _is_truthy(cls, value: object) -> bool: + """Return whether a metadata value is treated as truthy.""" + if value is None: + return False + if isinstance(value, bool): + return value + if isinstance(value, (int, float)) and not isinstance(value, bool): + return value != 0 + if isinstance(value, (list, tuple, set, dict)): + return len(value) > 0 + text = str(value).strip().lower() + if text in cls._FALSY_TOKENS: + return False + return text in cls._TRUTHY_TOKENS or bool(text) + + @classmethod + def _wants_audio(cls, request: RouterRequest) -> bool: + """Return whether the request asks for audio support.""" + return ( + cls._is_truthy(request.metadata.get("requires_audio")) + or cls._is_truthy(request.metadata.get("audio_input")) + or cls._is_truthy(request.metadata.get("audio")) + ) + + @staticmethod + def _audio_allowlist(request: RouterRequest) -> frozenset[str] | None: + """Parse an optional ``metadata.audio_models`` allowlist.""" + raw = request.metadata.get("audio_models") + if raw is None: + return None + if isinstance(raw, str): + parts: Iterable[object] = raw.split(",") + elif isinstance(raw, Iterable) and not isinstance(raw, (bytes, bytearray)): + parts = raw + else: + return frozenset() + return frozenset(stripped.lower() for item in parts if (stripped := str(item).strip())) + + def _capabilities_for(self, model: str, request: RouterRequest) -> frozenset[str] | None: + """Resolve an explicit capability set, or ``None`` when absent.""" + overrides = request.metadata.get("model_capabilities") + if isinstance(overrides, Mapping) and model in overrides: + override = overrides[model] + if isinstance(override, str): + return frozenset( + stripped.lower() for part in override.split(",") if (stripped := part.strip()) + ) + if isinstance(override, Iterable) and not isinstance(override, (bytes, bytearray)): + return frozenset(str(item).strip().lower() for item in override) + return frozenset() + if model in self._capability_map: + return self._capability_map[model] + return None + + def _supports_audio(self, model: str, request: RouterRequest) -> bool: + """Return whether a model is treated as audio capable.""" + allowlist = self._audio_allowlist(request) + if allowlist is not None: + return model.lower() in allowlist + capabilities = self._capabilities_for(model, request) + if capabilities is not None: + return "audio" in capabilities + lower = model.lower() + return any(token in lower for token in self._NAME_TOKENS) + + def choose(self, request: RouterRequest, signals: TaskSignals) -> RoutingDecision: + """Prefer audio models when the capability is requested.""" + wants = self._wants_audio(request) + eligible = [ + candidate + for candidate in self._model_catalog.values() + if signals.domain_tag in candidate.supports_domains + ] or list(self._model_catalog.values()) + healthy = [ + candidate + for candidate in eligible + if self._provider_health.is_available(candidate.provider) + ] + active = healthy or eligible + costs = { + candidate.model: candidate.estimate_cost( + signals.prompt_tokens_estimate, + request.max_tokens, + ) + for candidate in active + } + audio_flags = { + candidate.model: self._supports_audio(candidate.model, request) for candidate in active + } + availability_note = "healthy" if healthy else "circuit-open emergency" + + if wants: + selected = max( + active, + key=lambda candidate: ( + audio_flags[candidate.model], + candidate.quality_score, + -costs[candidate.model], + candidate.model, + ), + ) + fallback_candidates = sorted( + (candidate for candidate in active if candidate.model != selected.model), + key=lambda candidate: ( + not audio_flags[candidate.model], + -candidate.quality_score, + costs[candidate.model], + candidate.model, + ), + ) + capable_note = "audio-capable" if audio_flags[selected.model] else "non-audio fallback" + rationale = ( + f"audio-input-prefer requested; selected {availability_note} " + f"{capable_note} {selected.model} (quality {selected.quality_score:.2f})" + ) + else: + selected = max( + active, + key=lambda candidate: ( + candidate.quality_score, + -costs[candidate.model], + candidate.model, + ), + ) + fallback_candidates = sorted( + (candidate for candidate in active if candidate.model != selected.model), + key=lambda candidate: ( + -candidate.quality_score, + costs[candidate.model], + candidate.model, + ), + ) + rationale = ( + f"audio-input-prefer no requires_audio/audio_input/audio signal; " + f"selected {availability_note} quality-first {selected.model}" + ) + + return RoutingDecision( + chosen_model=selected.model, + provider=selected.provider, + routing_strategy=self.strategy_name, + rationale=rationale, + fallback_chain=[candidate.model for candidate in fallback_candidates[:3]], + ) + + def build_strategies( model_catalog: Mapping[str, ModelCandidate], latency_stats: LatencyStats, @@ -12999,6 +13187,11 @@ def build_strategies( model_catalog=model_catalog, provider_health=provider_health, ), + RoutingStrategyName.AUDIO_INPUT_PREFER: AudioInputPreferStrategy( + model_catalog=model_catalog, + provider_health=provider_health, + capability_map=model_capability_map, + ), RoutingStrategyName.AB_TEST: ABRoutingStrategy( model_catalog, ab_model_a, diff --git a/tests/test_audio_input_prefer_strategy.py b/tests/test_audio_input_prefer_strategy.py new file mode 100644 index 0000000..db04e6d --- /dev/null +++ b/tests/test_audio_input_prefer_strategy.py @@ -0,0 +1,196 @@ +"""Tests for audio-input-prefer routing.""" + +from router.config import RouterSettings, default_model_catalog +from router.model_ids import ( + ANTHROPIC_SAFETY_MODEL, + GEMINI_PRO_MODEL, + MOONSHOT_BALANCED_MODEL, + OPENAI_FRONTIER_MODEL, +) +from router.schemas import ( + ChatMessage, + DomainTag, + LatencyRequirement, + RouterRequest, + RoutingStrategyName, + TaskSignals, +) +from router.strategies import ( + AudioInputPreferStrategy, + InflightStats, + LatencyStats, + SuccessStats, + build_strategies, +) +from safety.circuit_breaker import CircuitBreakerRegistry + + +class _FakeHealth: + def __init__(self, unavailable: set[str] | None = None) -> None: + self._unavailable = unavailable or set() + + def is_available(self, provider: str) -> bool: + return provider not in self._unavailable + + +def _signals() -> TaskSignals: + return TaskSignals( + complexity_score=0.5, + domain_tag=DomainTag.GENERAL, + latency_requirement=LatencyRequirement.BATCH, + token_budget=4096, + prompt_tokens_estimate=128, + ) + + +def _request(metadata: dict | None = None) -> RouterRequest: + return RouterRequest( + request_id="req-audio-input-prefer", + messages=[ChatMessage(content="Transcribe this clip.")], + metadata=metadata or {}, + ) + + +def _strategy( + unavailable: set[str] | None = None, + capability_map: dict[str, frozenset[str]] | None = None, +) -> AudioInputPreferStrategy: + return AudioInputPreferStrategy( + default_model_catalog(), + _FakeHealth(unavailable), + capability_map=capability_map, + ) + + +def test_audio_input_prefer_enum_parses() -> None: + assert RoutingStrategyName("audio-input-prefer") is RoutingStrategyName.AUDIO_INPUT_PREFER + + +def test_audio_input_prefer_quality_first_when_absent() -> None: + decision = _strategy().choose(_request(), _signals()) + + assert decision.chosen_model == ANTHROPIC_SAFETY_MODEL + assert "quality-first" in decision.rationale + assert decision.routing_strategy is RoutingStrategyName.AUDIO_INPUT_PREFER + + +def test_audio_input_prefer_falsy_flags_stay_quality_first() -> None: + decision = _strategy().choose( + _request({"requires_audio": False, "audio_input": "no", "audio": 0}), + _signals(), + ) + + assert decision.chosen_model == ANTHROPIC_SAFETY_MODEL + assert "quality-first" in decision.rationale + + +def test_audio_input_prefer_requires_audio_prefers_capable() -> None: + catalog = default_model_catalog() + capability_map = {model: frozenset({"tools"}) for model in catalog} + capability_map[MOONSHOT_BALANCED_MODEL] = frozenset({"audio"}) + decision = _strategy(capability_map=capability_map).choose( + _request({"requires_audio": True}), _signals() + ) + + assert decision.chosen_model == MOONSHOT_BALANCED_MODEL + assert "audio-capable" in decision.rationale + + +def test_audio_input_prefer_audio_input_alias() -> None: + catalog = default_model_catalog() + capability_map = {model: frozenset({"tools"}) for model in catalog} + capability_map[MOONSHOT_BALANCED_MODEL] = frozenset({"audio"}) + decision = _strategy(capability_map=capability_map).choose( + _request({"audio_input": "yes"}), _signals() + ) + + assert decision.chosen_model == MOONSHOT_BALANCED_MODEL + + +def test_audio_input_prefer_audio_alias() -> None: + catalog = default_model_catalog() + capability_map = {model: frozenset({"tools"}) for model in catalog} + capability_map[MOONSHOT_BALANCED_MODEL] = frozenset({"audio"}) + decision = _strategy(capability_map=capability_map).choose(_request({"audio": 1}), _signals()) + + assert decision.chosen_model == MOONSHOT_BALANCED_MODEL + + +def test_audio_input_prefer_audio_models_allowlist() -> None: + decision = _strategy(capability_map={}).choose( + _request( + { + "requires_audio": True, + "audio_models": [MOONSHOT_BALANCED_MODEL], + } + ), + _signals(), + ) + + assert decision.chosen_model == MOONSHOT_BALANCED_MODEL + assert "audio-capable" in decision.rationale + + +def test_audio_input_prefer_name_heuristic_when_map_absent() -> None: + decision = _strategy(capability_map={}).choose(_request({"requires_audio": True}), _signals()) + + # Empty map → name heuristic; gemini matches. + assert decision.chosen_model == GEMINI_PRO_MODEL + assert "audio-capable" in decision.rationale + + +def test_audio_input_prefer_respects_model_capabilities_override() -> None: + catalog = default_model_catalog() + decision = _strategy(capability_map={model: frozenset({"tools"}) for model in catalog}).choose( + _request( + { + "requires_audio": True, + "model_capabilities": {MOONSHOT_BALANCED_MODEL: "audio"}, + } + ), + _signals(), + ) + + assert decision.chosen_model == MOONSHOT_BALANCED_MODEL + + +def test_audio_input_prefer_known_map_audio_capability() -> None: + # Default known map marks gpt-5.5 / gemini* as audio-capable; gpt-5.5 leads quality. + decision = _strategy().choose(_request({"requires_audio": True}), _signals()) + + assert decision.chosen_model == OPENAI_FRONTIER_MODEL + assert "audio-capable" in decision.rationale + + +def test_audio_input_prefer_skips_unhealthy_providers() -> None: + decision = _strategy(unavailable={"openai", "google"}).choose( + _request({"requires_audio": True}), _signals() + ) + + assert decision.provider not in {"openai", "google"} + + +def test_audio_input_prefer_registered_by_strategy_factory() -> None: + settings = RouterSettings() + strategies = build_strategies( + default_model_catalog(), + LatencyStats(), + InflightStats(), + settings.quality_floor, + settings.ab_model_a, + settings.ab_model_b, + settings.ab_model_a_weight, + CircuitBreakerRegistry(), + settings.blend_quality_weight, + settings.blend_cost_weight, + settings.blend_latency_weight, + settings.request_cost_ceiling_usd, + settings.canary_stable_model, + settings.canary_model, + settings.canary_weight, + settings.latency_sla_ms, + success_stats=SuccessStats(), + ) + + strategy = strategies[RoutingStrategyName.AUDIO_INPUT_PREFER] + assert isinstance(strategy, AudioInputPreferStrategy)