Amazon scrapped an AI recruiting engine that penalized the word "women's"

Amazon's Edinburgh engineering hub spent four years building a recruiting engine that scored job applicants from one to five stars, much as shoppers rate products. By 2015 the team had found the problem. The models were trained on ten years of résumés submitted to the company, most of them from men, and the system had taught itself that male candidates were preferable. It penalized résumés containing the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. It also preferred verbs more common on male engineers' résumés, such as "executed" and "captured".

The group built 500 models for specific job functions and locations, each taught roughly 50,000 terms found on past applicants' résumés. Editing out the offending words was no guarantee the machines wouldn't devise other discriminatory ways of sorting candidates, and bad training data meant unqualified people were recommended for all manner of jobs, with results returning "almost at random". Amazon disbanded the team by the start of 2017. It kept a "much watered-down version" of the engine to cull duplicate candidate profiles from databases, and has since formed a new Edinburgh team to try automated employment screening again, this time with a focus on diversity.