Skip to content

Add copilot calibration measurement - #21

Merged
JuneQQQ merged 1 commit into
mainfrom
feature/copilot-calibration
May 18, 2026
Merged

Add copilot calibration measurement#21
JuneQQQ merged 1 commit into
mainfrom
feature/copilot-calibration

Conversation

@JuneQQQ

@JuneQQQ JuneQQQ commented May 18, 2026

Copy link
Copy Markdown
Owner

Summary

This cycle's competitive research → innovative increment.

Research. The WOLF benchmark (arXiv:2512.09187, Dec 2025) scores
werewolf-suspicion calibration with the Brier score. deepwolf is — in the
surveyed literature — the only werewolf project with an explainable
probabilistic copilot. So it is uniquely able to do something no competitor
can: report its own advisor's calibration back to the human who relies on it.

Shipped. A deepwolf calibrate command and deepwolf.copilot.calibration:

  • plays many seeded games; at each daybreak, asks the copilot — from every
    surviving villager's seat — for its suspicions, and pairs each with ground
    truth (was that player really a werewolf);
  • reports the Brier score (exact), the Brier skill score (vs a
    base-rate forecaster), the Murphy decomposition (reliability / resolution
    / uncertainty) and a reliability diagram.

Sample run (60 games, 7 players): Brier 0.206 vs baseline 0.226 — skill 0.087;
reliability 0.001 (the copilot is well-calibrated in the mid-range: it says 33%
and 34% turn out wolves, says 50% and 50% do) but slightly overconfident above
60%. The tool surfaces exactly that kind of finding.

Checklist

  • ruff / mypy / pytest (81 tests, 14 new) all pass
  • No behaviour change to existing code — purely additive
  • CHANGELOG.md and README.md updated
  • Research cited in the module docstring and the changelog

Notes for reviewers

Predictions are collected via the engine's observer hook at DAY_BREAKS — no
engine change needed. Only village-aligned viewpoints are scored (the copilot's
deductive job is a villager's job). The Murphy decomposition is the binned
approximation; the headline Brier score is computed exactly from the raw pairs.

Competitive research turned up the WOLF benchmark (arXiv:2512.09187), whose
metrics suite scores werewolf-suspicion calibration with the Brier score.
deepwolf is the only surveyed werewolf project with an explainable
probabilistic copilot — so it is the only one that can report its own
advisor's calibration back to the human relying on it.

- new deepwolf/copilot/calibration.py: evaluate_copilot plays seeded games,
  collects the copilot's suspicions from every villager's viewpoint at each
  daybreak, pairs them with ground truth, and scores them.
- CalibrationReport: exact Brier score, Brier skill score, the Murphy
  decomposition (reliability / resolution / uncertainty) and a reliability
  diagram; render() and to_markdown().
- new `deepwolf calibrate` CLI command with a --markdown export.
- 14 new tests; calibration helpers exported from the package root.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@JuneQQQ

JuneQQQ commented May 18, 2026

Copy link
Copy Markdown
Owner Author

Code review (post-merge record).

  • Research-driven and properly scoped: WOLF uses the Brier score for agent suspicion calibration; this adapts it to deepwolf's distinctive asset — the explainable copilot — which is a genuine extension, not a copy. ✅
  • Purely additive: no existing module is touched, predictions are harvested through the existing observer hook at DAY_BREAKS, so no game, seed or transcript changes. ✅
  • The metrics are sound: Brier is computed exactly from raw pairs; baseline Brier equals base_rate*(1-base_rate) (verified algebraically — that is exactly a base-rate forecaster's Brier); the Murphy decomposition is correctly labelled as the binned approximation. The decomposition test asserts the reconstruction to a tight tolerance. ✅
  • Only village-aligned viewpoints are scored — correct, since a werewolf trivially knows its packmates and would inflate calibration. ✅
  • Empty-report path is handled and tested; skill_score guards division by zero. ✅
  • CI green on 3.10-3.12, 81 tests.

The sample run is genuinely informative — it shows the copilot is well-calibrated mid-range but overconfident above 60%, which is a real, actionable finding. Merged via squash.

@JuneQQQ
JuneQQQ merged commit fa1f4de into main May 18, 2026
4 checks passed
@JuneQQQ
JuneQQQ deleted the feature/copilot-calibration branch May 18, 2026 17:10
JuneQQQ added a commit that referenced this pull request May 19, 2026
Follow-ups from a review of the copilot-calibration code (#21):

- the reliability diagram now shows each bin's calibration gap (predicted
  minus observed) in render(), to_markdown() and the CLI table — it is the
  most informative number per bin and CalibrationBin.gap already computed it
  but nothing displayed it.
- _score now builds the CalibrationReport in one place: n_games / n_players
  are threaded in rather than patched onto the object by the caller.
- evaluate_copilot's docstring notes that calibration is measured against
  games played by agent_factory, so results depend on the agents used.

Co-authored-by: JuneQQQ <june1243134432@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant