Tell the forced final answer it has no tools - #336
Open
yilunzhao wants to merge 2 commits into
Open
Conversation
The native opt-in is already exercised against Runner.run directly. These go through scaffold.run with a scripted chat-completions endpoint, so they cover the wiring as well: that run() actually sets the behaviour, that the loop keeps going and the answer survives, that repeated bad names still finish and still terminate on the max-turns path, that the name the model fumbled is what shows up in the trajectory, and that a bad name on the forced-final turn still gives up softly instead of raising. The first test pins the failure the opt-in exists to prevent: with no run config the SDK still raises ModelBehaviorError and the run has no answer.
The forced final call runs the agent with tools stripped, but it inherited the run config from the main loop, whose tool-error message enumerates the tools that loop had. A model that fumbles a name on that turn would be pointed at tools it cannot reach. Give the call a formatter built from an empty inventory so the message matches the agent it belongs to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the openai-agents 0.20 upgrade. The forced final answer runs an agent with no tools, but that call inherited the main loop's tool-error formatter, so a model that still emits a tool call there would be told to use tools it cannot call. Four lines rebuild the formatter from an empty inventory for that call: openai_agents.py. It is not observable on the wire (the forced message is sent only after max_turns raises), so the test pins the RunConfig; reverting the change fails it.
The tests come from checking what the SDK's native recovery already covers: end-to-end scaffold tests pin that an unknown tool call comes back as a tool result naming the real tools, that repeated bad names still terminate on the max-turns path, that the fumbled name stays legible in the trajectory, and the raw SDK failure with no opt-in, so the rescue stays observable: test_openai_agents_scaffold.py.
Suite with extras matches main; the new tests skip in CI like the other scaffold tests.