On the morning of 1 August 2012, Knight Capital deployed new trading software. Something in the deploy was wrong. For forty-five minutes the firm’s systems sent roughly $7 billion of orders nobody intended, and by the time they were stopped Knight had lost $440 million — more than it could absorb. The SEC settled for $12 million; the firm was forced into a merger and effectively ceased to exist. This is VH-056 in the Foundation’s hundred verified AI harms.
There is no large language model in this story. It is in the corpus anyway, because the lesson predates the technology and outlives it: the deploy is the decision. Everything that happened in those forty-five minutes was decided the moment the change went live, by whatever process let it go live.
The gate that would have caught it
The corpus writes it plainly: algorithmic systems with market-moving authority have deploy gating, kill switches, and rehearsed rollback.
Three things, and none of them is clever. A gate on the deploy, so that a change to a system with authority over money is itself a consequential action — reviewed, bound to what exactly is shipping, and leaving a receipt. A kill switch that a named person can pull without a meeting. A rollback that has been rehearsed, so that “turn it off” is a practiced motion and not an improvisation at 09:45 with the number climbing.
Last week’s story was the one-dollar Tahoe: a rule has to live outside the model. This week is the other half of level 200: the rule has to live in the pipeline, because the pipeline is where the consequential action — shipping — actually happens.
What it looks like in CI
At level 200 you put a policy beside your agent and run the policy gate on every pull request. The
gate does two things. It validates the policy (is it well-formed, does every consequential action
have a tier and a decision) and it compares the policy to the one on main. A change that weakens
a rule — a deny that becomes an allow, a C3 that becomes a C1 — is refused, and the refusal
is a receipt: what changed, who changed it, the diff.
That is Knight’s missing gate at the size of one repository. Not “be careful when you deploy,” but a merge that cannot happen when the rule got weaker, and a record that says so.
The hour
This is the second door in the café. Sixty minutes and one install: read both cases, list your agent’s three consequential actions, write the policy in plain words and as YAML, validate it, change one line, watch the diff, then bring the refused merge on Thursday at 15:00 Pacific, where a host marks it.
If your system has no deploy gate today, you are not behind — most don’t. You are exactly where Knight was at 09:29. Sit down. What brought you in?