feat: configurable reasoning endpoint + provider regression test suite#3
Conversation
Pins the README happy-path before the provider config is refactored, so any regression is caught: clean-env defaults, documented model/env overrides, the OpenRouter media call contract (TTS/image/video + per-beat fallbacks), pure render helpers, the missing-key gates, and the SDK runtime patches. No real network calls — providers are faked and media bytes synthetic; ffmpeg-dependent tests skip if the binaries are absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Installs ffmpeg, pins Python 3.11 to match the Dockerfile, lints the test suite, and runs the full pytest suite. Lint scope is tests/ for now; src/ carries pre-existing ruff debt tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…I_KEY
Reasoning .ai() calls now resolve their key and base URL from optional env
vars, falling back to OpenRouter when unset. REEL_AF_API_KEY falls back to
OPENROUTER_API_KEY; an empty REEL_AF_API_BASE (Docker ${VAR:-}) falls back
to the OpenRouter default rather than clobbering it. Purely additive — with
both vars unset the flow is byte-identical to before. Media (TTS/image/
video) still routes through OpenRouter, so OPENROUTER_API_KEY stays
required. New vars plumbed through docker-compose and .env.example; tests
lock the override, precedence, fallback, and empty-string cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an advanced section near the bottom of the README explaining how to point reasoning at an OpenAI-compatible endpoint (local vLLM/Ollama, a self-hosted gateway), framed as the non-default path, with the two caveats (media stays on OpenRouter; use the openai/ prefix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
santoshkumarradha
left a comment
There was a problem hiding this comment.
LGTM. Verified locally beyond the unit suite:
Unit suite — uv run --extra dev pytest tests/ -q → 33/33 pass.
Wire-level E2E (local mock) — stood up an in-process OpenAI-compatible HTTP server, set REEL_AF_API_BASE + REEL_AF_API_KEY at it, ran app.ai(...). Mock received POST /v1/chat/completions with Authorization: Bearer <byo-key> and the configured model on the body. OpenRouter was never contacted. Proves litellm actually honors the override (the real risk the unit tests can't see).
Real-network E2E — pointed REEL_AF_API_BASE at https://openrouter.ai/api/v1 with the openai/ prefix (the README-documented BYO recipe) and a cheap model. Got a real response back. The override codepath works against a live OpenAI-compatible endpoint, not just against a fake.
Zero-config regression — cleared all BYO vars, only OPENROUTER_API_KEY set. Resolves to openrouter/deepseek/deepseek-v4-pro + https://openrouter.ai/api/v1. Byte-identical to pre-PR.
Non-blocking nits for a follow-up:
- Add a symmetric empty-string test for
REEL_AF_API_KEY=""(mirror of the existing empty-API_BASEtest). - Consider adding a wire-level mock E2E like the one above to
tests/— no network, deterministic, catches "litellm silently dropped api_base" regressions. - Worth one test pinning the
openai/prefix requirement (or a runtime warning when api_base is set with a non-openai/prefix). - Make sure issue #2 follow-up actually lifts the hard
OPENROUTER_API_KEYgate for pure-BYO users.
Tier A scope is honest, characterization-first ordering is the right call. Ship it.
Summary
Addresses #2 (first step). Lets users point reasoning at their own OpenAI-compatible endpoint via env vars, while keeping the OpenRouter default byte-identical. Lands behind a new characterization test suite so the original flow is provably unaffected.
This is Tier A of the plan discussed in the issue — reasoning only. Media (TTS/image/video) still routes through OpenRouter (tracked as follow-up).
Changes
Characterization test suite (the safety net, added first)
CI
tests/, runs the full suite on push/PR.Tier A — configurable reasoning endpoint (additive)
REEL_AF_API_BASE/REEL_AF_API_KEYoverride the reasoning base URL and key.REEL_AF_API_KEYfalls back toOPENROUTER_API_KEY; an emptyREEL_AF_API_BASE(Docker${VAR:-}) falls back to the OpenRouter default rather than clobbering it.docker-compose.ymland.env.example.Docs
openai/prefix).Test plan
uv run --extra dev python -m pytest tests/ -q→ 33 passed locally.uv run --extra dev ruff check tests/→ clean.Notes for reviewer
src//scripts/(11 issues) is intentionally not touched here; CI lint is scoped totests/. Happy to clean it in a separate PR.🤖 Generated with Claude Code