Skip to content

Surface LLM agent reasoning into the event log - #24

Merged
JuneQQQ merged 1 commit into
mainfrom
feature/agent-reasoning-trace
May 20, 2026
Merged

Surface LLM agent reasoning into the event log#24
JuneQQQ merged 1 commit into
mainfrom
feature/agent-reasoning-trace

Conversation

@JuneQQQ

@JuneQQQ JuneQQQ commented May 20, 2026

Copy link
Copy Markdown
Owner

Summary

Competitive research (this cycle) turned up the WOLF benchmark (arXiv:2512.09187), which makes per-agent chain-of-thought scratchpads central to explainable LLM social-deduction evaluation. deepwolf's LLMAgent already collected a reasoning_log per decision — but it was never exposed. Saved transcripts recorded what each agent did, never why.

This PR closes that gap.

What it does

  • New EventType.AGENT_REASONING, private to the deciding agent (visible_to = {actor}). Other agents' views are byte-for-byte unchanged.
  • Agent.last_reasoning() is a concrete default returning None; LLMAgent overrides it. Baseline agents (Random, etc.) emit nothing — no noise for non-LLM games.
  • engine._emit_reasoning() is called after each choice action: kill, inspect, protect, vote, shoot. Empty / whitespace reasoning is dropped.
  • Bilingual: a new agent_reasoning i18n key (en + zh). Chinese games render the reasoning prefix in Chinese.

deepwolf's twist over WOLF

WOLF keeps reasoning in out-of-band scratchpads. deepwolf threads it into the event log itself, so the reasoning is rendered, replayed and round-tripped through the same machinery as every other event — including the JSON transcript and the upcoming deepwolf replay (#23). The visibility model means an agent's reasoning never leaks into another agent's view.

Checklist

  • ruff / mypy / pytest (89 tests, 6 new) all pass
  • No change to existing event types — purely additive
  • English event text unchanged (no existing message touched)
  • CHANGELOG and README updated

Competitive research turned up the WOLF benchmark (arXiv:2512.09187), which
makes per-agent chain-of-thought scratchpads central to explainable LLM
social-deduction evaluation. deepwolf's LLMAgent already collected a
reasoning_log per decision but never exposed it — saved transcripts recorded
*what* each agent did, not *why*.

- new EventType.AGENT_REASONING, private to the deciding agent (visible_to
  only the actor) — other agents' views are byte-for-byte unchanged.
- Agent.last_reasoning() concrete default returns None; LLMAgent overrides
  it. RandomAgent and friends emit nothing.
- engine._emit_reasoning is called after each choice action (kill / inspect /
  protect / vote / shoot); empty or whitespace reasoning is dropped.
- bilingual: a new agent_reasoning i18n key (en + zh).
- 6 new tests; the transcript JSON now round-trips reasoning per decision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JuneQQQ

JuneQQQ commented May 20, 2026

Copy link
Copy Markdown
Owner Author

Code review (post-merge record).

  • Visibility is correct: AGENT_REASONING is restricted to visible_to={actor}, and verified by a dedicated test. Other agents' views (and therefore the copilot's belief model) are unchanged. ✅
  • Concrete-method opt-in: Agent.last_reasoning() returns None by default, so baseline agents emit no events — no noise for RandomAgent-only games. Confirmed by test_random_agent_emits_no_reasoning_events. ✅
  • Whitespace guard works at both layers (LLMAgent strips internally, engine strips defensively). Test test_empty_reasoning_string_is_not_emitted exercises the engine's guard. ✅
  • Bilingual: new agent_reasoning i18n key; Chinese smoke run shows "推理(kill):..." correctly. The decision-kind label (kill/vote/...) is left as a structural string — matches how phase is stored. ✅
  • The transcript round-trip via to_json carries reasoning end-to-end, so deepwolf replay (feat: add deepwolf replay command #23) will naturally surface it once that PR lands. Nice composition with another open PR. ✅
  • CI green on 3.10–3.12, 89 tests.

Merged via squash.

@JuneQQQ
JuneQQQ merged commit 01eadac into main May 20, 2026
4 checks passed
@JuneQQQ
JuneQQQ deleted the feature/agent-reasoning-trace branch May 20, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant