In 2023 a personal-injury case against the airline Avianca was heading through the federal court in Manhattan. The plaintiff’s lawyers, from the firm Levidow, Levidow & Oberman, needed case law to oppose a motion to dismiss. They used ChatGPT to find it, and it obliged: a brief with six citations, each with a case name, a court, a date and a quotation.
None of the six cases existed. The quotations were invented too.
Avianca’s lawyers could not find the cases and said so. The judge ordered the plaintiff’s lawyers to produce them. Here is the part worth remembering. Rather than go to a law library or a legal database, the lawyer asked ChatGPT whether the cases were real. It said they were. He filed that reassurance with the court.
What the corpus says happened next
Judge Castel sanctioned the lawyers $5,000 (Mata v. Avianca, S.D.N.Y., 678 F. Supp. 3d 443) and ordered them to write to every judge who had been falsely named as the author of a fake opinion. The court was misled, the other side was put to needless work, and real judges were credited with words they never wrote.
The model did what models do. It produced fluent, plausible text. The failure was that the text went from a chat window to a court filing with nobody opening a single case.
The gate that would have caught it
The corpus states it in one line: every consequential AI output is independently verified against the system of record before it is used or filed.
Two words carry the weight. Independently: a second question to the same model is not a check, it is the same guess again with more confidence. The check has to come from somewhere the model cannot reach, here a legal database or the court’s own records. System of record: for a citation that is the reporter itself; for a policy it is the policy; for a price it is the price list. Every output has one, and the gate is knowing where it lives.
Level 200 is where you start if you are already running it
Level 200 is the retrofit. The tool is already in use, the work already goes out under your name, and nobody wrote the rule down. It asks of one workflow: which outputs leave the building, what is the system of record for each, and who opens it before they do?
For the lawyers the honest answers were: all of them; the case law; and no one. Adding that one step, a person opening six cases, would have taken minutes.
The hour
This is the second door in the café: we’re already using AI and need to put a gate around it. Sixty minutes on paper. Pick one workflow where AI output goes to someone outside your team, list what it produces, name the source of truth for each item, and write down who checks it and when. Then bring it on Thursday at 15:00 Pacific, where a host marks it.
Do not ask the bot whether the bot is right. Sit down. What brought you in?