Skip to content

feat: configurable reasoning endpoint + provider regression test suite#3

Merged
santoshkumarradha merged 4 commits into
mainfrom
feat/byo-reasoning-endpoint
Jun 3, 2026
Merged

feat: configurable reasoning endpoint + provider regression test suite#3
santoshkumarradha merged 4 commits into
mainfrom
feat/byo-reasoning-endpoint

Conversation

@AbirAbbas

Copy link
Copy Markdown
Contributor

Summary

Addresses #2 (first step). Lets users point reasoning at their own OpenAI-compatible endpoint via env vars, while keeping the OpenRouter default byte-identical. Lands behind a new characterization test suite so the original flow is provably unaffected.

This is Tier A of the plan discussed in the issue — reasoning only. Media (TTS/image/video) still routes through OpenRouter (tracked as follow-up).

Changes

Characterization test suite (the safety net, added first)

  • 29 tests locking the documented happy path: clean-env defaults, model/env overrides, the OpenRouter media call contract (TTS/image/video + per-beat fallbacks), pure render helpers, missing-key gates, and the SDK runtime patches.
  • No real network calls — providers faked, media bytes synthetic; ffmpeg-dependent tests skip if the binaries are absent.

CI

  • New workflow: installs ffmpeg, pins Python 3.11 (matches the Dockerfile), lints tests/, runs the full suite on push/PR.

Tier A — configurable reasoning endpoint (additive)

  • REEL_AF_API_BASE / REEL_AF_API_KEY override the reasoning base URL and key.
  • REEL_AF_API_KEY falls back to OPENROUTER_API_KEY; an empty REEL_AF_API_BASE (Docker ${VAR:-}) falls back to the OpenRouter default rather than clobbering it.
  • With both unset, behavior is identical to before. New vars plumbed through docker-compose.yml and .env.example.
  • 4 new tests lock override / precedence / fallback / empty-string cases.

Docs

  • New advanced README section on bringing your own reasoning endpoint, framed as the non-default path, with caveats (media stays on OpenRouter; use the openai/ prefix).

Test plan

  • uv run --extra dev python -m pytest tests/ -q33 passed locally.
  • uv run --extra dev ruff check tests/ → clean.
  • Zero-config (no new env vars) regression covered by the baseline suite.

Notes for reviewer

  • Pre-existing ruff debt in src//scripts/ (11 issues) is intentionally not touched here; CI lint is scoped to tests/. Happy to clean it in a separate PR.

🤖 Generated with Claude Code

AbirAbbas and others added 4 commits June 2, 2026 17:06
Pins the README happy-path before the provider config is refactored, so
any regression is caught: clean-env defaults, documented model/env
overrides, the OpenRouter media call contract (TTS/image/video + per-beat
fallbacks), pure render helpers, the missing-key gates, and the SDK
runtime patches. No real network calls — providers are faked and media
bytes synthetic; ffmpeg-dependent tests skip if the binaries are absent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Installs ffmpeg, pins Python 3.11 to match the Dockerfile, lints the test
suite, and runs the full pytest suite. Lint scope is tests/ for now; src/
carries pre-existing ruff debt tracked separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…I_KEY

Reasoning .ai() calls now resolve their key and base URL from optional env
vars, falling back to OpenRouter when unset. REEL_AF_API_KEY falls back to
OPENROUTER_API_KEY; an empty REEL_AF_API_BASE (Docker ${VAR:-}) falls back
to the OpenRouter default rather than clobbering it. Purely additive — with
both vars unset the flow is byte-identical to before. Media (TTS/image/
video) still routes through OpenRouter, so OPENROUTER_API_KEY stays
required. New vars plumbed through docker-compose and .env.example; tests
lock the override, precedence, fallback, and empty-string cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an advanced section near the bottom of the README explaining how to
point reasoning at an OpenAI-compatible endpoint (local vLLM/Ollama, a
self-hosted gateway), framed as the non-default path, with the two caveats
(media stays on OpenRouter; use the openai/ prefix).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@santoshkumarradha santoshkumarradha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Verified locally beyond the unit suite:

Unit suiteuv run --extra dev pytest tests/ -q → 33/33 pass.

Wire-level E2E (local mock) — stood up an in-process OpenAI-compatible HTTP server, set REEL_AF_API_BASE + REEL_AF_API_KEY at it, ran app.ai(...). Mock received POST /v1/chat/completions with Authorization: Bearer <byo-key> and the configured model on the body. OpenRouter was never contacted. Proves litellm actually honors the override (the real risk the unit tests can't see).

Real-network E2E — pointed REEL_AF_API_BASE at https://openrouter.ai/api/v1 with the openai/ prefix (the README-documented BYO recipe) and a cheap model. Got a real response back. The override codepath works against a live OpenAI-compatible endpoint, not just against a fake.

Zero-config regression — cleared all BYO vars, only OPENROUTER_API_KEY set. Resolves to openrouter/deepseek/deepseek-v4-pro + https://openrouter.ai/api/v1. Byte-identical to pre-PR.

Non-blocking nits for a follow-up:

  • Add a symmetric empty-string test for REEL_AF_API_KEY="" (mirror of the existing empty-API_BASE test).
  • Consider adding a wire-level mock E2E like the one above to tests/ — no network, deterministic, catches "litellm silently dropped api_base" regressions.
  • Worth one test pinning the openai/ prefix requirement (or a runtime warning when api_base is set with a non-openai/ prefix).
  • Make sure issue #2 follow-up actually lifts the hard OPENROUTER_API_KEY gate for pure-BYO users.

Tier A scope is honest, characterization-first ordering is the right call. Ship it.

@santoshkumarradha santoshkumarradha merged commit 80b40a3 into main Jun 3, 2026
1 check passed
@santoshkumarradha santoshkumarradha deleted the feat/byo-reasoning-endpoint branch June 3, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants