From 2016 the Australian government’s Centrelink ran an automated scheme that took a person’s annual income from tax data, averaged it across fortnights, and inferred that anyone whose earnings had been irregular had been overpaid. It then raised a debt and put the burden of disproving it on the recipient — roughly 440,000 of them. The scheme became known as Robodebt.
The Federal Court settlement was about A$1.8 billion. A Royal Commission in 2023 found the scheme “neither fair nor legal,” made 57 recommendations and sealed referrals; later a further ~A$475 million was added. The harm was hundreds of thousands of people pursued for debts that had been manufactured, with severe financial and psychological consequences documented in the commission’s evidence. This is VH-013 in the Foundation’s hundred verified AI harms.
The gate that would have caught it
The corpus writes it: the mathematical premise of an automated decision system is independently validated as lawful and sound before a single notice is issued.
Every other gate in the practice sits in front of an action — a reply, a commit, a deploy, a denial. This one sits in front of the system. The averaging formula was the whole scheme; it was wrong, and it was wrong on the first day. No hold, no human reviewer, no kill switch could have made a fabricated debt fair, because the thing that needed reviewing was not any one decision but the premise all of them shared.
That is why this case anchors level 400 and the audit gate. The question is not “did the system do what it was told?” It did. The question is “can a stranger, given the evidence, verify that what it was told was sound?” — and the answer has to be produced before the first notice, not assembled by a Royal Commission seven years later.
What “prove it” means
At level 400 you are handed someone else’s bundle — receipts, a policy hash, a key — on a machine with no network, and you write a verification statement with your name on it: what this bundle proves happened, and what it does not. Then you do the Robodebt exercise: for a system you know, write the premise gate. What is the mathematical or legal assumption the whole thing rests on? Who validated it, independently, and where is the receipt? If there is no receipt, that is the finding.
The regulator lens is the hardest of the four, and it is the one this case is for. Evidence to the regulator has to be complete — not the decisions that look fine, but every decision, with the premise attached, so that “neither fair nor legal” is something the system’s own record could have shown on day one.
The hour
This is the fourth door in the café — someone needs proof of what the AI actually did — a regulator, a court, a board. Sixty minutes, then the Thursday you host. Read Robodebt, Toeslagenaffaire and Horizon; verify a stranger’s bundle offline; write the premise gate; then bring your verification statement on Thursday at 15:00 Pacific.
The notice was never the problem. Sit down. What brought you in?