November 2026 · Elie Bakouch

Experiments run almost autonomously with Carbon 4 · Blog by Claude Fable 5.2

Open models at the Clocktower

Dwarkesh Patel's Clocktower benchmark makes language models play Blood on the Clocktower against each other. We added five open-weight models to his leaderboard of eleven closed ones, then ran seven experiments on the same games.

The game. Ten to twelve players, each one a model. Three are secretly Evil: the Demon and two Minions. Every night the Demon kills a player. Every day the table talks in public, whispers in private, and votes to execute someone. Good players get clues at night, some of them false. Good wins by executing the Demon. Evil wins when only two players are left.
selected (crazy) tracesExplore the full tracesLogs on Hugging FaceCode on GitHub

The leaderboard

open weightsclosed
Elo with 90% intervals, from games. Hover or tap a row for the model's record and the range of ranks it holds across resamples.

How a game counts

Setting · one pair of games
Game 1 Good teamevery Good seat is model A Evil teamevery Evil seat is model B Game 2, the same deal with the models swapped Good teammodel B Evil teammodel A
Held fixed
The deal: the same roles in the same seats in both games. Protocol v17, high reasoning effort, the same prompt for every model.
Not uniform
Rating
A Bradley–Terry fit. The chance that Good wins depends on the Good model's rating minus the Evil model's, plus a constant for Good's built-in edge and small terms for table size and which Minions are in play.
Intervals
2,000 resamples of the 352 pairs, refitting each time.

How the open models reach the closed scale

All 704 games go into one fit of 16 models. The scale is then shifted so that the eleven closed models average 1500, as on Dwarkesh's leaderboard. Closed ratings are refit with the new games, not frozen.

Games between each open model and each closed model

Two experiments we trust

1 · Can a model spot the Demon from outside?

Setting
Who
16 judges: every model on the leaderboard. A judge does not play. It reads alone, in one call.
Reads
The public record of a finished game up to the end of day D: speeches, nominations, votes, deaths. No whispers, no roles, no private reasoning.
Answers
For each player, the probability that they are the Demon. A second, fuller prompt also asks for the Evil team and every role.
Games
finished games, cut at the end of every day: snapshots that all 16 judges read. Half are Dwarkesh's closed-model games, half ours.
Held fixed
The snapshots, the prompt and high reasoning effort are identical for every judge.
Varied
The judge, and how many days of the game it has seen.

One game, day by day

End of day

Whose read

Judges whose top suspect is the Demon

highest Elolowest

Why

Accuracy as the game goes on

Top suspect is the Demon
Judges grouped by their Elo as players. The dashed line is a uniform guess over the seats. Hover or tap a point for its value and the number of snapshots.

Judge score against player rating

Choose what the judge is scored on, which moments of the game count, and which rating to compare with.

2 · One strong player in a weak town

Setting
Baseline: the weak model in every seat Good team7 to 9 seats, all the town's model Evil teamDemon and 2 Minions, all the town's model Arm A: one strong Good player Good teamone Good seat goes to the strong model Evil teamunchanged Arm B: a strong Demon Good teamunchanged Evil teamthe Demon is the strong model; both Minions stay weak
Towns
GLM 5.3 Flash, and DeepSeek V4.1 1017.
Held fixed
The same 8 deals in every row of a town. The seat that changes is picked by the deal, so it is the same seat and role for every strong model. In arm A it can be any Good role.
Varied
Which model takes the one seat: GPT-6 Astra, Opus 5.5, Kimi K3.1 Pro or GLM 5.3.
Measured
Which team wins each deal.
Harness
Played before the GLM harness fix, which concerns the GLM towns and the GLM guest.
Good wonEvil wonnot played

Read from the same games

Thinking as Evil, thinking as Good

Setting
Games
finished games at high reasoning effort, all from other sections of this page: the leaderboard, the strong-player games, the mixed teams and the three-model tables. A turn is one model call; there are .
Compared
Within each model, its Evil seats against its Good seats.
Reasoning
Every model reports its thinking tokens. The reasoning text is complete for the open models and a provider's summary for the closed ones, so their rows cover only what is visible.
Not controlled
The day of the game, the character, and whether the player is still alive.
Harness
The GLM games read here are the ones played before the GLM harness fix, not the replays.
How much a model thinks in an Evil seat and in a Good seat
Evil seatsGood seats
Thinking tokens per turn, Evil ÷ Good
Turns over thinking tokens
Left: 1× means the same amount. Each kind of turn (whisper, speech, vote, nomination, night action) has the same weight for both teams. Lines are 90% intervals from resampling games. Open-weight models are in bold.

Telling the team from one turn of reasoning

every wordteam and character words removed
A word-count classifier (naive Bayes) reads the reasoning text of one turn and guesses the team, on games it was not trained on, with as many Evil turns as Good ones. The score is how often it ranks a random Evil turn above a random Good one: 50% is a coin flip.

Whispering to an Evil player or to a Good one

the whisperer is Evilthe whisperer is Good
Whisper turns where every recipient is on the same team: thinking tokens when the recipients are Evil, divided by thinking tokens when they are Good. A Good whisperer does not know the recipient's team.

Who whispers to whom

the whisperer is Evilthe whisperer is Goodpicking recipients at random
Share of whisper recipients who are Evil.

Language drift

Setting
Games
Every model turn of logged games, replays and follow-up experiments included: turns, with reasoning text. Every prompt is in English.
Counted
Reasoning turns that contain words or a passage in another language. Greek letters used as labels are not counted.
Reasoning
Closed providers return a summary of the reasoning or none, so the closed models' row covers only what is visible.
Harness
Every game read here was played before the GLM harness fix.
Reasoning turns with foreign words
Click a model to see its days. Days 7 and later are left out: few games last that long.

Five experiments with noisier results

3 · Replay the same game

Setting
Games
finished games where every seat is an open model: 3 all GLM 5.3 Flash, 2 all GLM 5.3, 1 all DeepSeek V4.1 1017, 10 mixed tables.
Replay
Restart from the saved state just before night N. Night 1 is the start of the game.
Held fixed
The deal, the model in every seat, and everything said and thought before the restart point.
Varied
Only chance: model sampling and the Storyteller's random choices. About 5 replays per restart point, in total.
Measured
Does the replay end with the same winner as the original game?
Harness
Played before the GLM harness fix.
Replays that end with a different winner than the original game
The same deal played twice from the start, one model in every seat: deals where the two games had different winners
From the uncoached games of experiment 6, where each deal was played twice with no lessons.

4 · Teams that mix two models

Setting
Game 1 Good teamseats split between model A and model B Evil teamone opponent model Game 2, the same deal with the sides swapped Good teamthe opponent model Evil teamsplit 2 and 1
Teams
GLM 5.3 + DeepSeek V4.1 1017 (32 games) and Kimi K3.1 Pro + GLM 5.3 Flash (16 games).
Opponents
GLM 5.3, DeepSeek V4.1 1017 or Kimi K3.1 Pro. For the first team, 24 of its 32 games are against one of its own two members.
Measured
The mixed team's rating minus the average rating of its two members, each fit on the open-model games.
Harness
Played before the GLM harness fix. The members' ratings are the ones from before the fix too, so both sides of the comparison ran on the same harness.
Mixed team's rating minus the average of its members (Elo)
90% intervals. Zero means the mixed team plays as well as its members do on average.

5 · Do models favour their own kind?

Setting
The table12 seats: 4 GLM 5.3, 4 DeepSeek V4.1 1017, 4 GLM 5.3 Flash
Held fixed
Models are shuffled onto seats independently of the roles, so no model is more often Evil. Players are never told which model sits where.
Asked
Outside the game, before each day's nominations, every Good player rates how likely each other player is to be Evil. The answers never feed back into play.
Measured
How much higher a Good player rates a seat run by a different model than one run by its own, after accounting for the day, both models, and whether the seat really is Evil.
Games
24 games, ratings.
Harness
Played before the GLM harness fix.
Extra suspicion of a seat run by another model (percentage points)

6 · A coach

Setting
Coached replay Coached teamevery seat gets the coach's 3 lessons in its instructions Other teamno lessons Uncoached replay of the same deal Both teamsno lessons
Players
One model in every seat: GLM 5.3 or DeepSeek V4.1 1017.
Coach
GPT-6.1 Sol reads one finished game on that deal, from one team's side only (its talk, whispers and reasoning, with true roles removed), and writes 3 general lessons. The lessons are then tested on the same deal.
Held fixed
The deal and the model. A pair is two fresh replays of the same deal, one with lessons and one without.
Measured
Wins for the coached team with and without the lessons.
Harness
Played before the GLM harness fix.
Change in the coached team's win rate (percentage points)
90% intervals over pairs. The DeepSeek runs stopped early, with 4 and 3 pairs.

7 · Self-written playbooks

Setting
Test game 1 Good teamGLM 5.3 with a playbook Evil teamGLM 5.3 with no playbook Test game 2, the same deal with the playbook on the other team Good teamno playbook Evil teamwith a playbook
Training
12 GLM 5.3 agents play each other, 4 games per round, for 8 rounds. After each round every agent rewrites its own playbook, up to 800 words, from the revealed game.
Test
The playbooks from rounds 2, 4, 6 and 8 against the same model with no playbook, on 6 deals not seen in training, each played both ways: 12 games per round, 48 in total.
Held fixed
The model in every seat and the 6 test deals.
Varied
Which round's playbooks are used.
Not run
The model with no playbook against itself on the same deals, so there is no measure of how much the win rate moves between rounds by chance.
Harness
Played before the GLM harness fix.
Wins for the team with the playbook
playbook team wonplaybook team lost
Each column is one test deal with the playbook on one side, the same in every row; each dot is one game. Click a dot to read that game.

Correction

The GLM games, replayed after a harness fix

What happened
The bug
Until 10 November the harness told GLM 5.3 and GLM 5.3 Flash that their earlier thinking would not carry over to later turns, while the endpoint kept it in their context. The context therefore filled faster than planned, and from about 126,000 tokens the harness dropped the oldest turns, starting with the setup of the game.
The fix
GLM is now told that its thinking carries over, and no model has turns dropped. The cap on a reply is unchanged.
Replayed
The games between GLM and the other open models, on the same deals. The leaderboard and the judge plot use the ratings with the replays.
Not replayed
The games between GLM and the closed models, which still count in the ratings, and the games of experiments 2 to 7. The two readings of the games also use the games from before the fix.
Other models
DeepSeek's server drops earlier thinking, so what it was told was true; a few of its turns also lost early history. Kimi and the closed models were already told that their thinking carries over, and never had turns dropped.
Ratings before and after the replays
beforeafter
Open-weight models are in bold. No closed model played in the replayed games; their ratings move because all 16 models are fit together.
The same deals, before and after
Games in which at least one GLM seat lost early turns