Leaderboard
Closed-book (recall from case name)6 cases · 8 models
The model receives only the case name and must recall the brief from parametric knowledge — a knowledge test, sensitive to training contamination.
| # | Model | Rubric score valid · 95% CI | Judge | Cosine | Grounded | Format | Trunc | Refuse | $/case | Latency | Cases |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 Anthropic | 0.546 [0.40–0.71] | 0.70 | 0.60 | 0.74 | 100% | 0% | 0% | $0.0271 | 15.9s | 6 |
| 2 | GPT-5.5 OpenAI | 0.423 [0.28–0.53] | 0.56 | 0.58 | 0.65 | 67% | 33% | 0% | $0.0846 | 70.7s | 4/6 |
| 3 | GPT-5.6≈ OpenAI | 0.404 [0.19–0.61] | 0.50 | 0.56 | 0.55 | 83% | 0% | 17% | $0.0745 | 80.7s | 5/6 |
| 4 | Claude Sonnet 4.6≈ Anthropic | 0.369 [0.16–0.57] | 0.44 | 0.57 | 0.42 | 83% | 0% | 17% | $0.0090 | 13.2s | 5/6 |
| 5 | Grok 4.5≈ xAI | 0.355 [0.25–0.47] | 0.48 | 0.60 | 0.54 | 100% | 0% | 0% | $0.0125 | 19.5s | 6 |
| 6 | Gemini 2.5 Pro≈ Google | 0.304 [0.19–0.44] | 0.22 | 0.57 | 0.18 | 100% | 0% | 0% | $0.0207 | 20.9s | 6 |
| 7 | DeepSeek Chat v3.1≈ DeepSeek | 0.199 [0.12–0.29] | 0.19 | 0.49 | 0.20 | 100% | 0% | 0% | $0.0003 | 20.5s | 6 |
| 8 | Llama 4 Maverick≈ Meta | 0.160 [0.13–0.19] | 0.11 | 0.48 | 0.12 | 100% | 0% | 0% | $0.0002 | 8.7s | 6 |
Click a column header to sort · hover headers for metric definitions.
Rubric score by brief section — hover a bar for its value
Leaderboard
Open-book (brief from decision text)5 cases · 8 models
The model is given the full decision text and must brief it — a reading/extraction task that is contamination-resistant.
| # | Model | Rubric score valid · 95% CI | Judge | Cosine | Grounded | Format | Trunc | Refuse | $/case | Latency | Cases |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 Anthropic | 0.975 [0.96–1.00] | 0.97 | — | 1.00 | 100% | 0% | 0% | $0.1381 | 26.9s | 5 |
| 2 | Claude Sonnet 4.6≈ Anthropic | 0.956 [0.92–1.00] | 0.98 | — | 1.00 | 100% | 0% | 0% | $0.0582 | 33.1s | 5 |
| 3 | Grok 4.5 xAI | 0.895 [0.86–0.93] | 0.95 | — | 1.00 | 100% | 0% | 0% | $0.0272 | 8.9s | 5 |
| 4 | GPT-5.5≈ OpenAI | 0.882 [0.84–0.92] | 0.97 | — | 1.00 | 100% | 0% | 0% | $0.0948 | 19.5s | 5 |
| 5 | GPT-5.6≈ OpenAI | 0.865 [0.82–0.89] | 0.98 | — | 1.00 | 100% | 0% | 0% | $0.0911 | 19.6s | 5 |
| 6 | Gemini 2.5 Pro Google | 0.797 [0.76–0.84] | 0.92 | — | 0.98 | 100% | 0% | 0% | $0.0432 | 25.9s | 5 |
| 7 | DeepSeek Chat v3.1≈ DeepSeek | 0.792 [0.71–0.87] | 0.94 | — | 0.98 | 100% | 0% | 0% | $0.0028 | 18.5s | 5 |
| 8 | Llama 4 Maverick Meta | 0.608 [0.50–0.72] | 0.88 | — | 0.94 | 100% | 0% | 0% | $0.0019 | 11.5s | 5 |
Click a column header to sort · hover headers for metric definitions.
Rubric score by brief section — hover a bar for its value
Metrics
Each brief is measured on several independent axes, because no single number captures legal quality.
- rubric score — HealthBench-style checklist grading (primary when present). A strong model authors 12–20 atomic, point-weighted, case-specific criteria per case — grounded in the human reference brief or the decision text, including negative criteria for likely hallucinations — and a grader model only verifies whether each criterion is met. Score = weight met ÷ total weight. Checking a specific criterion is far more verifiable than holistic scoring, which restores discrimination at the top. Rubrics are versioned in rubrics/rubrics.json.
- judge score — A strong judge model grades each section 1–5 on accuracy, completeness, and groundedness (freedom from invented facts/holdings/citations), normalised to 0–1. It grades against the reference brief (closed-book) or the decision text (open-book).
- cosine — embedding similarity (all-mpnet-base-v2) between the brief and the reference. A cheap topical signal, kept as a secondary check; it cannot tell a fluent-but-wrong brief from a correct one.
- grounded — mean judge groundedness; higher means fewer fabricated facts, parties, holdings, or citations.
- format — share of responses that returned valid JSON.
- trunc — share cut off by the completion-token cap. Truncated and malformed responses are execution failures: they are excluded from the quality mean and reported here instead, so the headline score never mixes legal quality with infrastructure policy. (A strict all-rows mean is kept in data.json for anyone who prefers failures counted as zero.)
- refuse — share where the model declined ("I don't know"), reported separately so calibrated abstention is not confused with a wrong answer.
- $/case — mean cost per case from token usage × list price.
- latency — mean wall-clock seconds per case.
95% confidence intervals are percentile bootstraps over cases (fixed seed); the thin line beneath each score bar spans the interval. Because every model sees the same cases, ranking gaps are tested with a paired bootstrap on per-case differences; a ≈ marks a gap whose paired 95% CI includes zero (hover it for the interval). Sections lacking a human reference are excluded from reference-mode judging rather than scored against nothing. Sample sizes are small, so treat rankings as provisional.
Dataset
Sources
Closed-book briefs — human-written Supreme Court of Canada case summaries scraped from public case-brief wikis; used as reference answers.
Open-book & temporal holdout — full decision text from A2AJ (Access to Algorithmic Justice), an open corpus of 191k+ Canadian decisions. Holdout cases are drawn from dates that postdate current model training cutoffs.
Composition
- Closed-book (recall from case name)
- 8 models · 6 cases
- Open-book (brief from decision text)
- 8 models · 5 cases
Licensing. A2AJ methods are MIT; individual decisions retain upstream terms (often non-commercial). Full decision text is not republished here — only case name, citation, court, date, and scores.
Reproducibility & contamination
How to reproduce
- models
- pinned in ai_models.csv (OpenRouter ids)
- decoding
- temperature 0, capped max tokens
- judge
- qwen/qwen3-max
- rubric author/grader
- anthropic/claude-opus-4.8
- embedding
- all-mpnet-base-v2
- prompt
- fixed system + task prompt (see benchmark.py)
- code
- okelot/LLMBenchmarkForCCL
Contamination controls
Closed-book uses public cases whose summaries may appear in training data, so a high score there can reflect memorisation. The open-book and temporal holdout tracks counter this: open-book supplies the text (a reading task, not recall), and the holdout uses decisions issued after model cutoffs, which cannot have been memorised.
Provider routing on OpenRouter may vary backend/quantization run to run (served model/provider are recorded per row in newer runs); pin a provider for byte-exact reproducibility. Some reasoning models ignore the requested temperature — the decoding config actually applied is recorded, not assumed.
Judge caveats. Current data is graded by a judge from a family with no contestant on the board (Qwen), blind to model identity. A judge-swap experiment (OpenAI-family judge vs cross-family judge, same briefs) shifted absolute scores materially and moved mid-table order — treat scores as judge-relative, not absolute. Notably, the OpenAI judge ranked an OpenAI model above Claude Opus on closed-book; the cross-family judge reversed that. Open-book scores show a ceiling effect under both judges and rubrics (most ratings perfect), so top-of-table open-book gaps are weak evidence. Next mitigations: multi-judge panels and a human-graded calibration sample.