Surface LLM agent reasoning into the event log - #24
Merged
Conversation
Competitive research turned up the WOLF benchmark (arXiv:2512.09187), which makes per-agent chain-of-thought scratchpads central to explainable LLM social-deduction evaluation. deepwolf's LLMAgent already collected a reasoning_log per decision but never exposed it — saved transcripts recorded *what* each agent did, not *why*. - new EventType.AGENT_REASONING, private to the deciding agent (visible_to only the actor) — other agents' views are byte-for-byte unchanged. - Agent.last_reasoning() concrete default returns None; LLMAgent overrides it. RandomAgent and friends emit nothing. - engine._emit_reasoning is called after each choice action (kill / inspect / protect / vote / shoot); empty or whitespace reasoning is dropped. - bilingual: a new agent_reasoning i18n key (en + zh). - 6 new tests; the transcript JSON now round-trips reasoning per decision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
Code review (post-merge record).
Merged via squash. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Competitive research (this cycle) turned up the WOLF benchmark (arXiv:2512.09187), which makes per-agent chain-of-thought scratchpads central to explainable LLM social-deduction evaluation. deepwolf's
LLMAgentalready collected areasoning_logper decision — but it was never exposed. Saved transcripts recorded what each agent did, never why.This PR closes that gap.
What it does
EventType.AGENT_REASONING, private to the deciding agent (visible_to = {actor}). Other agents' views are byte-for-byte unchanged.Agent.last_reasoning()is a concrete default returningNone;LLMAgentoverrides it. Baseline agents (Random, etc.) emit nothing — no noise for non-LLM games.engine._emit_reasoning()is called after each choice action: kill, inspect, protect, vote, shoot. Empty / whitespace reasoning is dropped.agent_reasoningi18n key (en + zh). Chinese games render the reasoning prefix in Chinese.deepwolf's twist over WOLF
WOLF keeps reasoning in out-of-band scratchpads. deepwolf threads it into the event log itself, so the reasoning is rendered, replayed and round-tripped through the same machinery as every other event — including the JSON transcript and the upcoming
deepwolf replay(#23). The visibility model means an agent's reasoning never leaks into another agent's view.Checklist
ruff/mypy/pytest(89 tests, 6 new) all pass