Benchmark lab
AI Judge
One bundle. Three independent judges. Reproducible rankings.
A single-operator instrument for honest model comparison — streamed answers, deterministic validators, seeded blind judge panels, durable SQLite records.
Methodology
Bundle
One immutable, versioned prompt bundle — 1–5 typed tasks. Same input for every model, forever.
Stream
Candidates answer live over OpenRouter. Deterministic validators check JSON, counts, word limits and known answers first.
Judge ×3
A seeded blind panel of three judges scores each answer at temperature 0. Candidate identity is never revealed.
Rank
Median of judge overalls per task, then averaged across tasks. Scored runs enter the board, including incomplete ones with penalties. Cancelled runs never do.
Current standings
coding-sample
| Rank | Model | Median | Runs | Last evaluated |
|---|---|---|---|---|
| — | ~anthropic/claude-opus-latestPROVISIONAL | 10.0 | 1 | 19d ago |
| — | deepseek/deepseek-v4-pro-0813PROVISIONAL | 9.9 | 1 | Aug 28, 2026 |
| — | openai/gpt-5.6-luna-proPROVISIONAL | 9.9 | 1 | Aug 28, 2026 |
| — | ~anthropic/claude-sonnet-latestPROVISIONAL | 9.8 | 1 | 19d ago |
| — | deepseek/deepseek-v4-flash-0731PROVISIONAL | 9.8 | 1 | Aug 28, 2026 |
Why it is honest
Blind judging
Judge prompts contain only the task and the raw answer — never the model's name, provider, or metadata.
Seeded panels
Each category gets one deterministic 3-judge panel per run, persisted with its seed and reserve order. Fully reproducible.
Deterministic validators
Objective checks — schema, counts, word limits, known math answers — run before judging and are shown separately, never blended away.