Leaderboard
Closed-book (recall from case name)50 cases · 8 models
The model receives only the case name and must recall the brief from parametric knowledge — a knowledge test, sensitive to training contamination.
| # | Model | Rubric score valid · 95% CI | Judge | Cosine | Grounded | Format | Trunc | Refuse | $/case | Latency | Cases |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 Anthropic | 0.633 [0.58–0.68] | 0.79 | 0.65 | 0.83 | 94% | 0% | 6% | $0.0261 | 17.2s | 47/50 |
| 2 | GPT-5.6 OpenAI | 0.543 [0.48–0.60] | 0.71 | 0.62 | 0.75 | 84% | 0% | 8% | $0.0518 | 57.8s | 42/50 |
| 3 | Grok 4.5≈ xAI | 0.502 [0.45–0.55] | 0.62 | 0.63 | 0.65 | 98% | 0% | 2% | $0.0119 | 22.1s | 49/50 |
| 4 | Claude Sonnet 4.6 Anthropic | 0.468 [0.40–0.53] | 0.57 | 0.62 | 0.57 | 92% | 0% | 8% | $0.0098 | 17.3s | 46/50 |
| 5 | GPT-5.5≈ OpenAI | 0.467 [0.42–0.52] | 0.70 | 0.61 | 0.77 | 96% | 0% | 4% | $0.0763 | 65.9s | 48/50 |
| 6 | Gemini 2.5 Pro≈ Google | 0.464 [0.40–0.53] | 0.48 | 0.61 | 0.47 | 100% | 0% | 0% | $0.0211 | 22.1s | 50 |
| 7 | DeepSeek Chat v3.1 DeepSeek | 0.293 [0.25–0.34] | 0.31 | 0.55 | 0.32 | 100% | 0% | 0% | $0.0004 | 22.4s | 50 |
| 8 | Llama 4 Maverick Meta | 0.208 [0.17–0.25] | 0.23 | 0.53 | 0.25 | 98% | 0% | 0% | $0.0002 | 9.3s | 49/50 |
Click a column header to sort · hover headers for metric definitions.
Rubric score by brief section — hover a bar for its value
Leaderboard
Open-book (brief from decision text)20 cases · 8 models
The model is given the full decision text and must brief it — a reading/extraction task that is contamination-resistant.
| # | Model | Rubric score valid · 95% CI | Judge | Cosine | Grounded | Format | Trunc | Refuse | $/case | Latency | Cases |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.8 Anthropic | 0.974 [0.96–0.98] | 0.96 | — | 0.98 | 100% | 0% | 0% | $0.1247 | 25.2s | 20 |
| 2 | Claude Sonnet 4.6 Anthropic | 0.934 [0.92–0.95] | 0.97 | — | 0.99 | 100% | 0% | 0% | $0.0540 | 30.9s | 20 |
| 3 | Grok 4.5≈ xAI | 0.914 [0.89–0.94] | 0.93 | — | 0.97 | 100% | 0% | 0% | $0.0258 | 9.7s | 20 |
| 4 | GPT-5.5≈ OpenAI | 0.901 [0.88–0.93] | 0.95 | — | 0.98 | 100% | 0% | 0% | $0.0861 | 19.3s | 20 |
| 5 | GPT-5.6≈ OpenAI | 0.888 [0.85–0.92] | 0.95 | — | 0.98 | 100% | 0% | 0% | $0.0817 | 18.3s | 20 |
| 6 | Gemini 2.5 Pro Google | 0.778 [0.75–0.80] | 0.93 | — | 0.98 | 90% | 0% | 0% | $0.0381 | 35.9s | 18/20 |
| 7 | DeepSeek Chat v3.1 DeepSeek | 0.703 [0.66–0.74] | 0.90 | — | 0.95 | 100% | 0% | 0% | $0.0025 | 22.7s | 20 |
| 8 | Llama 4 Maverick Meta | 0.609 [0.57–0.65] | 0.86 | — | 0.93 | 100% | 0% | 0% | $0.0018 | 12.1s | 20 |
Click a column header to sort · hover headers for metric definitions.
Rubric score by brief section — hover a bar for its value
Metrics
Each brief is measured on several independent axes, because no single number captures legal quality.
- rubric score — HealthBench-style checklist grading (primary when present). A strong model authors 12–20 atomic, point-weighted, case-specific criteria per case — grounded in the human reference brief or the decision text, including negative criteria for likely hallucinations — and a grader model only verifies whether each criterion is met. Score = weight met ÷ total weight. Checking a specific criterion is far more verifiable than holistic scoring, which restores discrimination at the top. Rubrics are versioned in rubrics/rubrics.json.
- judge score — A strong judge model grades each section 1–5 on accuracy, completeness, and groundedness (freedom from invented facts/holdings/citations), normalised to 0–1. It grades against the reference brief (closed-book) or the decision text (open-book).
- cosine — embedding similarity (all-mpnet-base-v2) between the brief and the reference. A cheap topical signal, kept as a secondary check; it cannot tell a fluent-but-wrong brief from a correct one.
- grounded — mean judge groundedness; higher means fewer fabricated facts, parties, holdings, or citations.
- format — share of responses that returned valid JSON.
- trunc — share cut off by the completion-token cap. Truncated and malformed responses are execution failures: they are excluded from the quality mean and reported here instead, so the headline score never mixes legal quality with infrastructure policy. (A strict all-rows mean is kept in data.json for anyone who prefers failures counted as zero.)
- refuse — share where the model declined ("I don't know"), reported separately so calibrated abstention is not confused with a wrong answer.
- $/case — mean cost per case from token usage × list price.
- latency — mean wall-clock seconds per case.
95% confidence intervals are percentile bootstraps over cases (fixed seed); the thin line beneath each score bar spans the interval. Because every model sees the same cases, ranking gaps are tested with a paired bootstrap on per-case differences; a ≈ marks a gap whose paired 95% CI includes zero (hover it for the interval). Sections lacking a human reference are excluded from reference-mode judging rather than scored against nothing. Sample sizes are small, so treat rankings as provisional.
Dataset
Sources
Closed-book briefs — human-written Supreme Court of Canada case summaries scraped from public case-brief wikis; used as reference answers.
Open-book & temporal holdout — full decision text from A2AJ (Access to Algorithmic Justice), an open corpus of 191k+ Canadian decisions. Holdout cases are drawn from dates that postdate current model training cutoffs.
Composition
- Closed-book (recall from case name)
- 8 models · 50 cases
- Open-book (brief from decision text)
- 8 models · 20 cases
Licensing. A2AJ methods are MIT; individual decisions retain upstream terms (often non-commercial). Full decision text is not republished here — only case name, citation, court, date, and scores.
Reproducibility & contamination
How to reproduce
- models
- pinned in ai_models.csv (OpenRouter ids)
- decoding
- temperature 0, capped max tokens
- judge
- qwen/qwen3-max
- rubric author/grader
- anthropic/claude-opus-4.8
- embedding
- all-mpnet-base-v2
- prompt
- fixed system + task prompt (see benchmark.py)
- code
- okelot/LLMBenchmarkForCCL
Contamination controls
Closed-book uses public cases whose summaries may appear in training data, so a high score there can reflect memorisation. The open-book and temporal holdout tracks counter this: open-book supplies the text (a reading task, not recall), and the holdout uses decisions issued after model cutoffs, which cannot have been memorised.
Provider routing on OpenRouter may vary backend/quantization run to run (served model/provider are recorded per row in newer runs); pin a provider for byte-exact reproducibility. Some reasoning models ignore the requested temperature — the decoding config actually applied is recorded, not assumed.
Judge caveats. Current data is graded by a judge from a family with no contestant on the board (Qwen), blind to model identity. A judge-swap experiment (OpenAI-family judge vs cross-family judge, same briefs) shifted absolute scores materially and moved mid-table order — treat scores as judge-relative, not absolute. Notably, the OpenAI judge ranked an OpenAI model above Claude Opus on closed-book; the cross-family judge reversed that. Open-book scores show a ceiling effect under both judges and rubrics (most ratings perfect), so top-of-table open-book gaps are weak evidence. Next mitigations: multi-judge panels and a human-graded calibration sample.