- Harshness
- -0.2 vs panel median — harsh-0.2
- Parse fails
- 40%first-try 6 · repaired 3 · invalid 5
- Variance
- 0.4Panel-wide σ 1.8
- Claim Δ
- 0.1
How reliable are the judges themselves?
Agreement is a diagnostic, not a target — a well-supported minority judgment is not penalized. Scores here affect judge meta-rating only and never alter candidate rankings.
| Model | Judgments | Harshness | Variance | Parse fails | Evidence | Claim Δ | |
|---|---|---|---|---|---|---|---|
| claude-sonnet-5anthropic | 10 | -0.2 | 0.4Panel-wide σ 1.8 | 40% | 5.7 | 0.1 | |
| gpt-5.6-luna-proopenai | 7 | -0.3 | 0.7Panel-wide σ 1.8 | 0% | 6.2 | 0.2 | |
| gpt-5.6-solopenai | 10 | +0.6 | 0.4Panel-wide σ 1.8 | 0% | 6.1 | 0.2 | |
| grok-4.5x-ai | 21 | -0.1 | 0.2Panel-wide σ 1.8 | 0% | 6.1 | 0.3 | |
| grok-4.6x-ai | 17 | -0.2 | 0.6Panel-wide σ 1.8 | 0% | 6.7 | 0.4 | |
| glm-5.3z-ai | 14 | +0.1 | 0.4Panel-wide σ 1.8 | 0% | 6.2 | 0.4 | |
| glm-5.3-flashz-ai | 14 | +0.1 | 0.2Panel-wide σ 1.8 | 7% | 6.1 | 0.4 | |
| gpt-luna-latest~openai | 3 | -0.2 | 0.2Panel-wide σ 1.8 | 0% | 5.6 | 0.1 | |
| gpt-sol-latest~openai | 3 | +0.2 | 0.2Panel-wide σ 1.8 | 0% | 5.3 | 0.0 | |
| grok-latest~x-ai | 3 | -0.2 | 0.1Panel-wide σ 1.8 | 0% | 6.8 | 0.1 |
Optional fixture check: did this judge score known answers the way a human reviewer expected? Separate from the rollups above, and unused for candidate rankings.
No fixture runs recorded yet.
Calibration sends a small set of human-reviewed answers through a judge to check whether it scores known-good and known-bad cases in the expected range. It is optional and never changes candidate rankings.