Thread by @ajc730

How many complaints against police officers can be abated by terminating a few bad apples? Jacob Kaplan and I consider this question in a new paper that is now published in @CPPJournal. We believe the answer is likely to be "not a lot." onlinelibrary.wiley.com/doi/10.1111/17…
Thread 👇

You've probably heard that a very small number of officers are responsible for a very large share of misconduct or citizen complaints. Estimates vary quite a bit but I've seen estimates like 2% of officers account for half of all use of force complaints.

The naive conclusion from calculations like this is that if only we could identify who these bad apple officers are, we could make a considerable dent in addressing police misconduct by terminating (or re-assigning) them.

Why is this conclusion naive? Because these are ex ante calculations which don't account for the difficulty of identifying bad apples in the first place or for a problem which we refer to as "data density bias."

By data density bias, we are referring to the fact that citizen complaints against officers are rare: far fewer than one complaint per officer per year. With rare events, it is tautological to claim that a small share of officers account for an outsize share of complaints.

For instance, if there are 10 complaints among 100 officers, even if those complaints all accrue to different officers, 10% of officers would account for 100% of the complaints. This doesn't necessarily mean that complaints are concentrated among a small number of officers.

How can we assess the effectiveness of identifying and terminating bad apples? We begin by noting that it is difficult to fire a police officer after the end of his or her probationary period, usually a period of around 18 months.

Using complaint or use of force data compiled from an officer's initial work history, can we identify who the bad apples will be? Sort of. As it turns out, there is persistence in generating complaints but 18 months doesn't provide a whole lot of information to make a prediction.

Using data from the Chicago PD, we identify officers with the greatest number of complaints in their probationary period and simulate their replacement with alternative officers. We conclude that firing the riskiest 2% of officers would reduce use of force complaints by ~ 2-3%.

Even terminating the 10% of riskiest officers, a proposition which would likely present huge political and management challenges for a police department, would have reduced UoF complaints by around 6-7%.

This computation does not account for the deterrence or spillover effects of firing bad apples. Perhaps terminating bad apples improves behavior amongst the remaining apples in the cart. If so, our estimates could be too small.

While we don't have a natural experiment to address this, a paper by Quispe-Torrealba and Stewart estimates the magnitude of spillover effects from officer movement in the London Met: nature.com/articles/s4156…

Taking their estimate at face value means that our estimates are 80% too small. That sounds like a big deal but the base rates are low. Terminating the top 2% of officers might then reduce use of force complaints by as much as 4-5% rather than 2-3%. Progress but not a panacea.

So how can we make progress? We have a few ideas: 1) Filing a complaint can be costly. As such, official complaint data is probably be very incomplete. Easier access to filing a complaint is likely to generate better data. See this paper by @bocar_a: assets.aeaweb.org/asset-server/f…

2) Extend officer probationary review periods. With a 5-year probationary period, it becomes easier to identify bad apples and therefore to incapacitate future misconduct.

Overall, as numerous scholars in this area have noted, in order to be effective, early warning systems must promote deterrence and accountability among a broader set of officers. Firing bad apples is necessary but not sufficient.

We want to point out that there is a reply essay to our paper by @michaelsierraa and @AVPapachristos that presents a different perspective on our analysis and findings. While we stand by our work, their response is definitely worth reading: onlinelibrary.wiley.com/doi/10.1111/17…

For those who want to get into the weeds, we have prepared an informal "response to their response" which can be found here: jacobdkaplan.com/documents/poli…