@PNASNews published a study last year claiming no racial bias in police shootings. The study's central claim was mathematically unsupported. @dean_c_knox & I submitted critique to PNAS, which was rejected. We appealed. Today PNAS published our critique.1/n pnas.org/content/117/3/…
The original study by Johnson et al. claimed to find “White officers are not more likely to shoot minority civilians than non-White officers.” i.e., the fatal shooting rate of minorities by white officers is ≤ that of nonwhite officers. 2/n 
In promoting their work, the authors went much further with their claims, stating “black officers were just as likely to shoot black citizens as white officers were” and “our findings show no support for the idea that white officers are biased in shooting black citizens” 3/n
These unsupported claims reached the halls of Congress, prompting @DrPhilGoff to correct the record (and school the House Judiciary committee on causal inference). 4/n
Since the study only analyzed fatal encounters, it couldn't possibly recover shooting rates. *Every observation* in the data was a shooting. No variation in outcomes. Instead, it estimated the probability that a fatally shot person was black, white, etc. 5/n
Using this approach the study concluded no racial bias simply because more fatally shot civilians were white (which we already knew from @washingtonpost). This is not a test of racial bias. For ex., there could be more fatally shot whites just bc they are the majority group. 6/n
By analyzing fatal shootings only, and ignoring the vast majority of other encounters which did not escalate to that level, officers who pulled the trigger in 5 out of 5 encounters, or in 5 out of 1000 encounters, would appear identical. 7/n
More formally, the study made claims about P(shot|race), but estimated P(race|shot). Those are *not* the same. That’s not a statement of preference. It’s a mathematical fact, Bayes’ rule, that we’ve known since the 1700s. 8/n
Bayes rules shows w/out knowing # of encounters w/ white/nonwhite officers, can't know which group more likely to shoot. If we *could* compute shooting rates we'd then ask whether encounters being compared were otherwise similar on relevant traits (i.e. no omitted variables). 9/n
In this case, no need to debate omitted variables. Original study estimated fundamentally incorrect quantity. And to maintain officer anonymity, study's posted data don't contain info on officer race, and there is no posted code. So the analyses cannot be verified/replicated.10/n
Johnson and Cesario have stated their analysis is still informative because it controls for county crime rates by race and other shooting attributes. But as Bayes’ rule shows, the addition of these control variables, X, does not solve the fundamental conceptual problem. 11/n
Racial bias in policing is one of the most pressing policy concerns of our time. Serious questions remain over how best to study it. We recognize that many academic debates live in a gray area, where both sides’ arguments have some merit. This is not one of those cases. (12/n)
Scientists, journal editors and peer reviewers must be able to distinguish genuine debates from provable errors in logic. Our credibility, and the ability to advance evidence-based policy, depends on it.
We hope scholars and policymakers benefit from our critique. (n/n)
We hope scholars and policymakers benefit from our critique. (n/n)
