Skip to content

feat(llm): pass reasoning effort through to reasoning models - #319

Open
Jacky628 wants to merge 1 commit into
aiming-lab:mainfrom
Jacky628:feat/reasoning-effort-passthrough
Open

Jacky628 wants to merge 1 commit into
aiming-lab:mainfrom
Jacky628:feat/reasoning-effort-passthrough

Conversation

@Jacky628

Copy link
Copy Markdown
Contributor

Reasoning models accept a reasoning.effort parameter, but there is currently no way to set it, so every call runs at the provider default.

This matters most for the judge roles added in #316. A reviewer or a tournament judge is exactly the kind of task where more deliberation changes the verdict rather than just the wording — it is a scoring decision, not a generation one.

llm.reasoning_effort (xhigh | high | medium | low | minimal | none) is threaded into both the generator and the reviewer client. Empty by default, so nothing changes unless it is set.

The part worth reviewing

The parameter is gated on the model, because non-reasoning models reject an unknown reasoning field with HTTP 400. Getting this wrong would break every run on a gpt-4o-class model, so:

  • _supports_reasoning_effort reuses the existing _NEW_PARAM_MODELS set instead of introducing a second model list. The models that need max_completion_tokens are the same reasoning family, and one list cannot drift out of sync with itself.
  • It tolerates a provider prefix, so the check still works for openai/gpt-5.4 through OpenRouter — the existing _NEW_PARAM_MODELS call sites do a bare startswith and would miss those.

One consequence to flag: o1 is not covered, because it is absent from _NEW_PARAM_MODELS. I did not add it, since that set also controls the token parameter and I did not want to change unrelated behaviour in this PR — but if o1 should be there, adding it fixes both call sites at once.

Verified

13 model names classify as expected, including provider-prefixed and non-OpenAI ones:

gpt-5.4, gpt-5, o3, o3-mini, o4-mini            -> True
openai/gpt-5.4, openai/o3-mini                  -> True
gpt-4o, gpt-4.1, claude-sonnet-4-6, deepseek-chat, ""  -> False
anthropic/claude-sonnet-4-6                     -> False

The example config still parses to an empty value, and a configured effort reaches the reviewer client built by reviewer_from_rc_config.

Full suite: 2951 passed, 0 failed, 56 skipped.

Independent of #317 and #318; it touches a different part of the example config, so the three should not conflict.

🤖 Generated with Claude Code

Reasoning models accept a `reasoning.effort` parameter, but there was no
way to set it, so every call ran at the provider default. The setting
matters most for the judge roles added in aiming-lab#316 — a reviewer or tournament
judge is exactly the kind of task where more deliberation changes the
verdict rather than just the wording.

`llm.reasoning_effort` (xhigh | high | medium | low | minimal | none) is
threaded into both the generator and the reviewer client. Empty by
default, so nothing changes unless it is set.

The parameter is gated on the model: non-reasoning models reject an
unknown `reasoning` field with HTTP 400, so they must never be sent it.
_supports_reasoning_effort reuses the existing _NEW_PARAM_MODELS set
rather than introducing a second model list — the models that need
max_completion_tokens are the same reasoning family, and one list cannot
drift out of sync with itself. It also tolerates a provider prefix, so
the check still works for "openai/gpt-5.4" through OpenRouter.

Note that this means o1 is not covered, because it is absent from
_NEW_PARAM_MODELS; adding it there would fix both call sites at once if
that is intended.

Verified: 13 model names classify as expected (including provider-prefixed
and non-OpenAI ones), the example config still parses to an empty value,
and a configured effort reaches the reviewer client.

Full suite: 2951 passed, 0 failed, 56 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant