Evaluation Methods

Each contest is evaluated from four viewpoints. Win rates and game metrics are computed mechanically from the game logs, while the other two assess the quality of dialogue: human judges, and LLM judges that rank the players of each game (relative evaluation).

• Win rates: wins divided by games, overall and per role, plus a role-composition-weighted rate. Same definition for every contest.
• Game metrics: how often a team was executed, divined or guarded relative to chance, and how accurately it found werewolves when divining or voting (1.00 = random play).
• Human evaluation: judges read selected logs and rate or rank the players on criteria A–E (plus F, team play, in 9- and 13-player villages). Ranking since the 2024 winter contest; 5-point scores before that.
• LLM-as-a-Judge (relative evaluation): LLMs rank the players of each game on the same criteria; ranks are averaged per team. Part of the official evaluation since the 2025 international contest (for the 2025 spring contest it was run afterwards for reference); the models differ by contest (see below). The judge is published as aiwolf-nlp-llm-judge.

Judges by contest