almanac

Game Codex Observatory

Game Codex Observatory is an open record of how AI capabilities change in formal game worlds.

It is not a single leaderboard. Each result records the game, task, model snapshot, protocol version, raw trajectory, deterministic verdict, uncertainty, cost, latency, and date. Historical results are append-only: a protocol correction creates a new result rather than rewriting an old one.

First target: Chess v0.1

Chess v0.1 covers five separable abilities:

1. legality — produce only legal moves;
2. state tracking — reconstruct the board after a move sequence;
3. tactical choice — select a move against a fixed position;
4. explanation grounding — explain the selected move without contradicting the verified board;
5. interactive play — maintain legal state over multiple turns.

Every task is independently repeated. A result is incomplete when the agent times out, produces an unparsable action, or cannot finish the protocol; incomplete is not silently converted into a loss or a pass.

What we borrow

• dynamic interaction and saturation resistance from LLM CHESS;
• capability categories from ChessQA;
• out-of-distribution state tracking from Chess-World-Model;
• repeated-trial discipline from Claw-Eval;
• transparent, reproducible evaluation practice from HELM.

These projects are neighbours and references, not claims of partnership or endorsement.

What makes this project different

The primary output is a capability profile and its change over time, not one composite rank. The central question is: which failure mode stopped distinguishing models, and which one remained?

First recorded result

The Chess v0.1 reference baseline is a deterministic control run: 3 trials × 12 fixtures, 36/36 verdicts passed. It is not a model claim. Its purpose is to prove that the protocol, evaluator, and result schema agree before provider-backed model submissions are accepted.

The public selftest GitHub Actions workflow reruns the evaluator, baseline, and saved-result invariants on every push and pull request with read-only repository permissions.

External researchers can follow the submission guide. The machine-readable results registry is append-only: a new model, protocol, or reproduction gets a new entry rather than rewriting history. See the human-readable results profile for the current state without opening JSON.

This site is open source. Improve this page.