Skip to content

Add the arena leaderboard - #12

Merged
JuneQQQ merged 1 commit into
mainfrom
feature/arena-leaderboard
May 18, 2026
Merged

Add the arena leaderboard#12
JuneQQQ merged 1 commit into
mainfrom
feature/arena-leaderboard

Conversation

@JuneQQQ

@JuneQQQ JuneQQQ commented May 18, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the arena leaderboard requested in #4.

A leaderboard ranks agents fairly. Each competitor is measured under
identical, seeded conditions:

  • it plays the werewolves against a fixed reference village, and
  • it plays the village against fixed reference werewolves.

Its score is the mean of the two win rates. Because both match-ups reuse the
same seeds for every competitor, the ranking is reproducible and the only
variable is the agent itself.

Closes #4

Changes

  • New deepwolf/arena/leaderboard.pyLeaderboard, LeaderboardReport
    (with render() and to_markdown()), LeaderboardEntry. It reuses the
    existing Arena for each match-up rather than duplicating the game loop.
  • New deepwolf leaderboard CLI command, with --markdown PATH to export a
    paste-ready table and --provider env / --model to add a real model as a
    competitor.
  • 7 new tests (ranking, reproducibility, Markdown, score arithmetic, progress).

Checklist

  • ruff check . passes
  • mypy deepwolf passes
  • pytest passes (50 tests; 7 new)
  • CHANGELOG.md and README.md updated

Notes for reviewers

A "competitor" is a faction-agnostic AgentBuilder (Callable[[int], Agent]),
so the same agent can be slotted into either side. Leaderboard._arena composes
two builders into the (player_id, role) -> Agent factory the Arena expects.

Ranks agents fairly: every competitor plays both the werewolves (against a
fixed reference village) and the village (against fixed reference werewolves)
under identical seeds. Score is the mean of the two win rates.

- new deepwolf/arena/leaderboard.py: Leaderboard, LeaderboardReport,
  LeaderboardEntry; reuses the existing Arena for each match-up.
- new `deepwolf leaderboard` CLI command with a --markdown export.
- 7 new tests.

Closes #4

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JuneQQQ

JuneQQQ commented May 18, 2026

Copy link
Copy Markdown
Owner Author

Code review (post-merge record).

  • Leaderboard reuses Arena rather than duplicating the game loop — both match-ups are just two Arena runs with composed factories. Good, no logic drift risk. ✅
  • Fairness holds: every competitor's two arenas use the same base_seed, so competitors face identical games and the score difference is purely the agent. ✅
  • _arena composes two faction-agnostic AgentBuilders into the (player_id, role) -> Agent factory — clean separation, and it means a real model added via --provider env drops in with no special handling. ✅
  • to_markdown() produces a paste-ready table; reproducibility is covered by a test. ✅
  • Empty competitor field is rejected early with a clear ValueError. ✅
  • CI green on 3.10–3.12.

Nice — this is what turns the arena into an actual benchmark. Merged via squash.

@JuneQQQ
JuneQQQ merged commit c4efe83 into main May 18, 2026
4 checks passed
@JuneQQQ
JuneQQQ deleted the feature/arena-leaderboard branch May 18, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Arena leaderboard: rank models from arena runs

1 participant