Can you trust AI with Canadian case law?

The results

Two tests, because "knowing the law" and "reading a judgment" are different skills.

Each model prepares a standard case brief — facts, issue, decision, reasons, and the ratio — and earns a percentage: the share of checklist points its brief got right. Think of it as a law-school grade, marked by a very consistent (if non-human) marker.

How to read a score

• 80%+Strong — reliably captures what the case decided. Still verify citations.
• 65–79%Good — sound overall, but expect gaps in the details.
• 50–64%Mixed — useful as a starting point, not as an answer.
• 30–49%Weak — significant errors or omissions. Treat with suspicion.
• Under 30%Poor — do not rely on it. Much of the brief is wrong or missing.

How we test

The same way you'd test an articling student.

1. Pick real Canadian decisions

   Cases come from public sources, including the A2AJ open corpus of Canadian court decisions. For each case we hold a trusted reference: a human-written brief, or the decision text itself.

2. Ask each AI for a standard case brief

   Facts, issue, decision, reasons, ratio — the same five-part brief every law student learns. Every model gets the identical instructions.

3. Mark it against a case-specific checklist

   Each case has a detailed marking checklist — did the brief name the right parties, the right outcome, the actual legal principle? The checklist also watches for things a model is likely to invent. Points earned ÷ points available = the score you see above.

4. Check for invented content separately

   Made-up facts, holdings, or citations are the single biggest risk of using AI in practice — Canadian courts have already dealt with filings containing non-existent cases. We track this separately, and flag it in the results.

Full disclosure: the marking is done by an AI applying the checklist — not by a panel of lawyers — and some test runs cover a small number of cases. Treat the numbers as a strong indication, not a certification. Every methodological detail, caveat, and raw score is in the researcher version.

What it means for your practice

Three rules of thumb the data supports.

• Never trust an AI's memory of a case. Scores on the "from memory" test are consistently far below the "judgment in hand" test. An AI answering from memory will sound just as confident while inventing parties, outcomes, and citations. If you only remember one thing from this page, make it this one.
• Give it the document. The same models score dramatically higher when you paste in the actual decision text. Used as a reading assistant — "summarize this judgment I'm giving you" — AI is far more dependable than as an oracle.
• Verify citations, every time. Even the best scores here are not 100%, and your professional obligations don't transfer to the software. Several Canadian law societies have issued guidance on generative AI; checking that every cited case exists and says what the AI claims remains your job.

Common questions

The fine print, in plain English.

Who runs this, and is anyone paying for the results?

LexBench is an independent open-source project. The code, the test cases, and the marking checklists are all public on GitHub, so anyone can re-run the tests or challenge the grading. No AI vendor sponsors it.

Why is an AI doing the marking?

Grading hundreds of briefs by hand doesn't scale, so a separate AI — from a different company than any model being tested — applies each case's checklist. Checking "does the brief state that the appeal was dismissed?" is a much more reliable task for an AI than open-ended grading. It's still a limitation, which is why the checklists are published and human spot-checking is on the roadmap.

Couldn't the AI have just memorized these cases?

For famous cases, yes — that's partly what the "from memory" test measures. That's also why the "judgment in hand" test matters more for practice: it measures reading, not recall. The benchmark also tests decisions issued after the models were trained, which they cannot have memorized.

Does a high score mean I can rely on the output?

No. A high score means the model made fewer mistakes on our test cases. It is not legal advice, not a product endorsement, and not a substitute for reading the authority yourself. Rules of professional conduct on competence and supervision apply to AI-assisted work the same way they apply to work delegated to a student.

What exactly do "facts, issue, decision, reasons, ratio" scores measure?

Each brief is scored section by section: did it get the material facts right, state the issue the court actually decided, report the correct decision, capture the court's reasons, and extract the ratio — the binding principle. In our results so far, models are strongest on the issue and outcome, and weakest on facts and ratio — exactly the parts you'd rely on most.