In 2018 Reuters reported on an experiment inside Amazon. A team had built a model to screen résumés: feed it the pile, and it hands back the best candidates, ranked. To teach it what “best” looked like, they trained it on ten years of résumés the company had actually received.
That history was overwhelmingly male. The model did what a model does with a pattern. It learned that being male looked like being hired, and it learned to penalize the word women’s, as in “women’s chess club captain”. It downgraded graduates of two women’s colleges.
What the corpus says happened next
Amazon scrapped the project. Engineers could not guarantee the model was neutral, and the company has said it was never used as the sole basis for a hiring decision. That is the good version of this story. The harm that would have followed, a screen that systematically ranked women lower, was caught inside the building, before it reached a single applicant.
Hold on to why it was caught. Nobody got hurt first. Somebody looked at what the model had learned before the model was allowed to rank people.
The gate
The corpus puts it in one line: bias from historical data is tested and mitigated before any screen ranks human beings.
Notice what the gate does not say. It does not say the data was bad on purpose, and it was not. It was a faithful record of who had been hired. The problem is that a faithful record of the past is an instruction to repeat it. The test has to be run on the model’s output, by someone who did not build it, and it has to happen before the first real applicant is ranked, not after the first complaint.
Level 100 is where you start if you have not started
Level 100 is for the person who has not yet put AI into the work and wants to do it with a gate in place from day one. It asks one small thing of one workflow: what is the decision, who does it affect, and what would we check before we let the tool influence it?
If your answer to the last part is “nothing yet”, that is the right place to be standing. You are earlier than Amazon’s engineers were, and the whole exercise fits in an hour.
The hour
This is the first door in the café: we are about to start using AI and want to do it properly. Pick one decision about people that you would like help with. Hiring, scheduling, who gets a callback. Write down what data the tool would learn from, what that data says about the past, and what test you would run on its output before it touched a real person. Then bring it on Thursday at 15:00 Pacific, where a host marks it.
Do not trust a ranking because it is tidy. Sit down. What brought you in?