There's No Undo in Court

You may be surprised to hear this, but I’m a bit of a legal nerd (the nerd part is the surprising part, obviously). I actually enjoyed my jury duty service a decade ago, including when I was selected for a weeklong trial. Recently, I was disappointed when my jury duty summons didn’t require me to report, because I wanted to see how the federal district court compared to the county court I served in before. I have been a regular listener of the Advisory Opinions podcast with Sarah Isgur and special guest David French for years. My analysis of a Supreme Court argument (yes, I listened to it live) got read on the pod. There are even times when my tech nerdery and legal nerdery come together, like when lawyers get caught using LLMs to write their briefs because of hallucinated case citations or quotes.

On a recent podcast, a discussion of yet more lawyers behaving badly with LLM-based tools caught my attention. Not so much the details of the incident, but why LLMs and the agents built on top of them still struggle when it comes to law. LLMs are not making significantly more mistakes (hallucinations, omissions, flawed arguments) in law when compared to other fields. But when mistakes happen, you can’t easily correct them, and the consequences for mistakes are high.

The conversation started with a discussion of a new LLM benchmark from the legal AI company Harvey. Benchmarks are essentially tests to see how well LLMs can perform on specific tasks. AI companies previously used actual tests, like the bar exam required to become a lawyer, but when LLMs started passing those consistently, they created new, dedicated benchmarks for LLMs. The new “Legal Agent Benchmark” was created to evaluate LLMs on “real world work done by lawyers.” It uses a strict evaluation method that they call “all-pass” grading, which doesn’t give partial credit for tasks that are “mostly correct.” As they say in the write-up: “A deal-team report that identifies eight of ten risks is not 80% useful; it is materially incomplete.” The goal is to apply the constraints of the real world: mistakes can’t be corrected and are costly.

So how do LLMs do on this test? In short — not well. As of this writing, Meta’s Muse Spark 1.1 gets the best grade at 20% (yes, it’s the same scale you’re used to from school). The much-discussed Claude Fable 5 from Anthropic scores 11.25%. When graded on the individual components of each task, top models score above 90%, but because every component has to be correct for the task to be correct, the all-pass scores are abysmal.¹

Scores like these didn’t surprise lawyers Sarah and David on the podcast, but they did surprise me, and I expect they surprised other folks in the AI or tech space. It’s a concrete example of how LLMs can seem incredibly capable (90% accuracy on components of legal work) but still be far from useful in the real world (20% accuracy on the whole task).

This gap between the capability of LLMs and agents and their actual usefulness for real-world tasks is one that I often see when folks in software interact with those who work in less digital types of jobs, like law. I find that folks in tech tend to overestimate the capabilities of LLMs in fields outside of software. Some of this is because software people are more likely to be tech optimists or AI boosters, but it’s true among the more pessimistic folks too.

Software wasn’t always so tolerant of mistakes. Before the internet was widespread, software was distributed on physical media like floppy disks, CDs, and game cartridges. “Shipping” your software meant actually shipping a physical copy of it to your customers or distributors. Once you put it on the disk, it was locked — bug fixes or improvements could only come in the next version a year later. The cost of mistakes was high: if the software was buggy, then people wouldn’t buy it — a massive financial loss. Patching critical issues in existing software meant creating new physical media and shipping it to customers experiencing the issue, incurring additional costs. This act of shipping physical media in the days before the internet is an example of what people in software call a “one-way door,” a door that, once you go through, you can’t go back. The term was popularized by Jeff Bezos, and generally speaking, the software industry tries to avoid them. Thankfully, the internet completely changed the way software was distributed.

These days, “shipping” software simply means adding a relatively small code change to a portion of a website. It’s all digital, and it happens in minutes. Instead of once a year, it happens multiple times a day. A suite of automated tests and multiple human beings have already reviewed and tested the code to make sure it doesn’t introduce any major bugs. Once it’s shipped, automated monitoring systems make sure that the change isn’t generating errors or causing key business metrics (order conversion rate, for example) to drop. If that happens, the change is “rolled back” automatically; the mistake is undone. Typically, changes are shipped first to a small percentage of users, so that if there is a problem it only impacts 5% or so of people using the site. Shipping software has gone from a “one-way door” to a “two-way door”: you can just walk right back through if something goes wrong. This is a best-case scenario: one-way doors still exist, especially in certain types of software. But in general the costs of mistakes have decreased significantly.

Lawyers can only dream of living in a world like that. The law is full of one-way doors: most mistakes can’t be undone. One of the many examples is the submission of briefs, formal documents that lay out the arguments for your position and against your opponent’s position. These are what got the lawyers I mentioned earlier in trouble: the LLM used to draft their filings cited hallucinated cases that set precedents in their favor. They didn’t catch these mistakes, but the judge did and ordered them to explain the citations. They submitted corrected versions, but the damage was already done. You can’t say, “Sorry, Your Honor, I realized an LLM hallucinated cases in my previous brief. Can I submit a correction and can we ignore that I violated the Federal Rules of Civil Procedure?” You’ve walked through that door, and you can’t walk back.

Making mistakes in such an environment is costly. All four lawyers were fined by the judge, two of them were banned from appearing before the Northern District of Mississippi for two years, and the judge cancelled the trial. This might not sound like a huge deal, but much of the legal profession depends on trust. Trust between clients and lawyers, trust between lawyers and judges. Once that trust goes away, it becomes much harder to practice law: lawyers can’t get hired if they can’t set foot in a federal courthouse, and clients don’t want to work with them if there’s a chance their trial gets cancelled.

The software world has built the technical infrastructure to mitigate the cost of mistakes as much as possible. If the code written by an LLM causes problems, the system that they’ve built will catch it and roll it back. But most of the world doesn’t work like this. “I’m sorry, Your Honor, we rolled back the code, so it’s all good now” doesn’t work in court.