Thread by @JanelleCShane

Very effective interactive demo of how human bias in training data affects any algorithm that tries to use it.

Especially in the US, predicting who will commit a crime is not the same as predicting who will be arrested. And here all the AI has is arrest data.

https://twitter.com/_KarenHao/status/1184799032721444865

I talk about this example in my book too. Neatly illustrates how if a machine learning algorithm tackles a problem like parole decisions without knowing about the bias in its data (and how can it? it doesn't know what's going on), it'll treat the bias as valid.

One solution to building fairer algorithms is to change the training data to reflect the world as it ought to be - in other words, to debias the data.

Here, one could adjust assuming a fair justice system would have produced equal arrest rates for Black vs white defendants.

The demo in the article shows how doing that might produce an algorithm that looks unfair when you look at its rearrest rates for Black vs white defendants - but that's only because the real-world arrest rate itself has bias built in.

As is often the case, it's so important to look carefully at what the AI is really being asked to predict.

Here, it's important to understand the difference, especially in the US, between the rates at which people commit crimes vs the rates at which they're arrested.

The algorithm doesn't know the difference. All it's doing is predicting human behavior based on the numbers it's given.

Making it predict the fair behavior we want, vs the biased behavior we have, requires careful work.

I talk more about the extreme literalmindedness & general obliviousness of machine learning algorithms in my book, You Look Like a Thing and I Love You: How AI Works and Why It's Making the World a Weirder Place janelleshane.com/book-you-look-…