Conversation
Reasoning models accept a `reasoning.effort` parameter, but there was no way to set it, so every call ran at the provider default. The setting matters most for the judge roles added in aiming-lab#316 — a reviewer or tournament judge is exactly the kind of task where more deliberation changes the verdict rather than just the wording. `llm.reasoning_effort` (xhigh | high | medium | low | minimal | none) is threaded into both the generator and the reviewer client. Empty by default, so nothing changes unless it is set. The parameter is gated on the model: non-reasoning models reject an unknown `reasoning` field with HTTP 400, so they must never be sent it. _supports_reasoning_effort reuses the existing _NEW_PARAM_MODELS set rather than introducing a second model list — the models that need max_completion_tokens are the same reasoning family, and one list cannot drift out of sync with itself. It also tolerates a provider prefix, so the check still works for "openai/gpt-5.4" through OpenRouter. Note that this means o1 is not covered, because it is absent from _NEW_PARAM_MODELS; adding it there would fix both call sites at once if that is intended. Verified: 13 model names classify as expected (including provider-prefixed and non-OpenAI ones), the example config still parses to an empty value, and a configured effort reaches the reviewer client. Full suite: 2951 passed, 0 failed, 56 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reasoning models accept a
reasoning.effortparameter, but there is currently no way to set it, so every call runs at the provider default.This matters most for the judge roles added in #316. A reviewer or a tournament judge is exactly the kind of task where more deliberation changes the verdict rather than just the wording — it is a scoring decision, not a generation one.
llm.reasoning_effort(xhigh | high | medium | low | minimal | none) is threaded into both the generator and the reviewer client. Empty by default, so nothing changes unless it is set.The part worth reviewing
The parameter is gated on the model, because non-reasoning models reject an unknown
reasoningfield with HTTP 400. Getting this wrong would break every run on agpt-4o-class model, so:_supports_reasoning_effortreuses the existing_NEW_PARAM_MODELSset instead of introducing a second model list. The models that needmax_completion_tokensare the same reasoning family, and one list cannot drift out of sync with itself.openai/gpt-5.4through OpenRouter — the existing_NEW_PARAM_MODELScall sites do a barestartswithand would miss those.One consequence to flag:
o1is not covered, because it is absent from_NEW_PARAM_MODELS. I did not add it, since that set also controls the token parameter and I did not want to change unrelated behaviour in this PR — but if o1 should be there, adding it fixes both call sites at once.Verified
13 model names classify as expected, including provider-prefixed and non-OpenAI ones:
The example config still parses to an empty value, and a configured effort reaches the reviewer client built by
reviewer_from_rc_config.Full suite: 2951 passed, 0 failed, 56 skipped.
Independent of #317 and #318; it touches a different part of the example config, so the three should not conflict.
🤖 Generated with Claude Code