Thread by @doctorow

When you or I seek out evidence to back up our existing beliefs and ignore the evidence that shows we're wrong, it's called "confirmation bias." 1/

It's a well-understood phenomenon that none of us are immune to, and thoughtful people put a lot of effort into countering it in themselves. 2/

If you'd like an unrolled version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

pluralistic.net/2021/12/02/emp… 3/

It's a well-understood phenomenon that none of us are immune to, and thoughtful people put a lot of effort into countering it in themselves. 4/

But confirmation bias isn't always an unconscious process. Consultancies like McKinsey have grown to multibillion-dollar titans by offering powerful people confirmation bias as a service. 5/

Pay them enough and they'll produce a fancy report saying whatever you *want* to do is the best thing you possibly *could* do.

pluralistic.net/2021/11/25/str… 6/

A sizable fraction of the #MachineLearning bubble is driven by this phenomenon. Pay a machine learning company enough money and they'll produce a statistical model that proves that whatever terrible thing you're doing is empirical and objective and true. 7/

After all, "Math doesn't lie."

The best term I've heard for this is "empirical facewash." I learned that term from @vm_wylbur, in a presentation on @hrdag's outstanding study of racial bias in predictive policing tools.

mic.com/articles/15628… 8/

(Sidenote: HRDAG just won the prestigious Rafto Prize, a major international award for human rights work)

hrdag.org/2021/09/23/hrd… 9/

(And on that note, HRDAG is a shoestring operation that turns our tax-deductible donations into reliable statistical accounts of human rights abuses that are critical to truth and reconciliation, human rights tribunals, and trials for crimes against humanity.) 10/

(I am an annual donor and you should consider them in your giving, too)

hrdag.networkforgood.com

Here's how Ball described that predictive policing research: everybody knows that cops have a racist policing problem, but we don't all agree on what that problem is. 11/

You and I might think that the problem is that cops make racially motivated arrests, while the cops and their apologists think the problem is that we *think* the cops make racially motivated arrests. 12/

By feeding crime data into a machine learning model, and then asking it to predict where crime will take place based on past patterns of crime data, cops can get an "objective" picture of where to concentrate their policing activities. 13/

But this has major problems. First, it presumes that crime stats are objective - that everyone reports crime at the same rate, and that the police investigate suspects at the same rate. 14/

In other words, this starts from the presumption that there is no racial bias in crime statistics - and then uses that presumption to prove that there is no racial bias in crime statistics! 15/

Second, this presumes that undetected crimes are correlated with detected ones. In other words, if the cops detect a lot of crime in a poor neighborhood - and not in a rich neighborhood - then all the undetected crimes are also in those poor neighborhoods. 16/

Finally, this presumes that every crime is a crime! In other words, it presumes that there are no de facto crimes like "driving while brown" or "walking your dog while black." 17/

Some of the "crimes" in the crime stats aren't actually crimes - rather, they're pretextual stops that turn into plea deals after bullying prosecutors threaten a long prison sentence. 18/

HRDAG's work crystallized the critique of machine learning as a tool for correcting systemic bias, and it has been my touchstone for understanding other bias-reinforcing/bias-accelerating machine learning scandals. 19/

It's a critical adjunct to such foundational texts as @mathbabedotorg's "Weapons of Math Destruction":

memex.craphound.com/2016/09/06/wea…

And @PopTechWorks's "Automating Inequality."

memex.craphound.com/2018/01/31/aut… 20/

But you don't need outside evidence that predictive policing reinforces and accelerates racial bias. Predpol's founders - leaders in predictive policing - came to the same conclusion in 2018, but decided not to do anything about it.

ieeexplore.ieee.org/abstract/docum… 21/

That may sound shocking, but really, it's par for the course with Predpol, which rebranded itself as @Geolitica_PS in order to distance itself from a string of scandals and bad publicity. 22/

From the start, Predpol has wrapped its operations in secrecy, pressuring police forces to hide their use of the service from city officials and residents. 23/

Back in 2018, a security researcher provided me with a list of cities that seemed to have secretly procured Predpol services.

boingboing.net/2018/10/30/el-… 24/

In 2019, @Motherboard's Caroline Haskins used that report to extract even more information about the cops' secret deals with Predpol:

vice.com/en/article/d3m… 25/

All this started because my source was able to learn that these cities were experimenting with Predpol's digital phrenology due to basic cybersecurity errors the company had made.

Predpol's cybersecurity has not improved since. 26/

A team of reporters from @gizmodo and @themarkup just published a blockbuster report on Predpol's role in biased policing, using the largest Predpol leak in history. Data regarding *5.9 million* Predpol predictions was left on an unsecured server!

themarkup.org/prediction-bia… 27/

The Markup/Gizmodo team used that dataset to conduct a massive study on racial bias in predictive policing. As American University's @ProfFerguson put it, "No one has done the work you guys are doing, which is looking at the data." 28/

This is "striking because people have been paying hundreds of thousands of dollars for this technology for a decade."

Ferguson's point - that public millions have been poured into an experimental technology without any external validation - is important. 29/

After all, it's unlikely that cops and Predpol keep this stuff a secret from us because they know we'll love it and they don't want to ruin the pleasant surprise. 30/

On the other hand, it makes perfect sense if Predpol is really selling empirical facewash - that is, confirmation bias as a service.

That certainly seems to be the case based on the analysis published today. 31/

Police who rely on Predpol do less patrolling in white and affluent neighborhoods (these are pretty much the same neighborhoods in most of the USA, of course). 32/

But when it comes communities of color and poor communities, Predpol predictions send cops flooding in: "A few neighborhoods in our data were the subject of more than 11,000 predictions." 33/

When Predpol sends cops into your neighborhood, arrests shoot up (you find crime where you look for it), as does use of force. 34/

This has knock-on effects - for example, the reporters tell the story of Brianna Hernandez, who was evicted, along with her two young children from low-income housing. Her partner was stopped in his car while dropping off some money for her. 35/

He had an old court injunction barring him from being on the premises because of a crime he committed 14 years earlier, while he was a minor. 36/

The housing complex had a policy of evicting tenants who associated with people who committed crimes, and it had been the target of a flood of Predpol predictions. 37/

Brianna doesn't know if she and her children were made homeless because of a Predpol prediction, thanks to the secrecy Predpol and its customers hide behind. 38/

Robert McCorquodale - the Calcasieu Parish, LA sheriff's attorney who handles public records requests - refused to confirm whether they used Predpol, despite the fact that the data-leak clearly confirmed they were. 39/

He cited "public safety and officer safety" and speculated that if criminals knew Predpol was in use, they'd be able to outwit it: "I feel this is not a public record." 40/

Unsurprisingly, police reform advocates in six of the cities where Predpol was in use didn't know about it: "Even those involved in government-organized social justice committees said they didn’t have a clue about it." 41/

All this secrecy helps hide the fact that Predpol is a) expensive and b) it doesn't work. But many police departments are wising up. 42/

LAPD was Predpol's first big reference customer - and they stopped using it in 2020, citing financial constraints and a damning Inspector General report.

It's not just LA. Santa Cruz - the birthplace of Predpol - also fired the company last year:

pluralistic.net/2020/06/27/bel… 43/

There's a limited pool of mathematicians who can produce the kind of convincing confirmation bias as a service that Predpol sells, and it's shrinking. 44/

1,400 mathematicians have signed an open letter "begging their colleagues not to collaborate on research with law enforcement, specifically singling out Predpol."

ams.org/journals/notic… 45/

Image:
Science Museum London (modified)
wellcomecollection.org/works/fjbypqcr…

CC BY 4.0
creativecommons.org/licenses/by/4.0

Cryteria (modified)
commons.wikimedia.org/wiki/File:HAL9…

CC BY 3.0:
creativecommons.org/licenses/by/3.… 46/