It was precrime for welfare: when you applied for benefits, it would assign you a risk score based on secretive algorithmic judgments that amounted to digital racism with empirical facewash: if you lived in a poor/racialized neighborhood, you were risky
But "algorithms don't lie" - that's right, they don't. They reflect their training data very well! If you have a human system that practices discrimination and use its data to train an automated system, it, too, will practice discrimination
But "the people the system identified were guilty": You can only find crime where you look for it. If you only ever spy on brown and poor people, you will only ever find crime among brown and poor people.
GIGO is the iron law of computing. Predictive policing systems predict the police, not crime.
Welfare systems around the world are rolling these things out, wooed by deep-pocketed, grifty AI hustlers who promise that they'll end accusations of bias forever, with a wink/nod that acknowledges that it's just the accusations they'll end, not the discrimination
"In the UK, as well as contracts with the outsourcing multinationals IBM, Tata Consultancy and Capgemini, the DWP is also working with UiPath, a New York-based company co-founded by Daniel Dines, the world’s first “bot billionaire"
