Our paper shows traditional analyses understate racial bias in police violence. A newly posted critique claims those approaches work great (if we assume away the problem). Given the bad science going around on the topic: thread 5harad.com/papers/post-tr… 
Our paper asks: given what we know about police-civilian interactions, how can we estimate racial bias using detainment records (stops/arrests) alone? We identify minimal assumptions, and show racial bias can be bounded using only data on stops. @dean_c_knox @conjugateprior
The challenge in this setting is that if officers are racially biased in stopping, then all records of police stops are "post-treatment," and incomplete police records make identifying the causal effect of civilian race on police behavior (e.g. use of force) much more difficult.
Instead of just worrying about controlling for common causes of treatment & outcome (omitted variable bias), conditioning on a mediator means other confounders can corrupt inferences. Especially unobserved common causes, U, of mediator & outcome. We state this explicitly.
This is a well-known selection problem in causal inference called "conditioning on a collider." By subsetting to stops, we open up new sources of confounding. We did not discover this. We merely clarified its implications in the policing context, and offered a potential solution.
The problem is especially worrisome in this context, because as we show, ignoring it (or mistakenly assuming it away) can lead to *drastic underestimates* of racial bias in police violence.
Despite this, the authors of this critique are not concerned with post-treatment selection bias (see their quote below). On this we fundamentally diverge. Our critics want to justify the same old statistical mistakes. We want to do better, and make the science better.
So in our paper, we ask: given both bias in stopping and unobserved confounders, U, what else would we need to estimate racial bias in police force using only data on stops? We thought this was a worthwhile scenario to explore.
We identify four assumptions that are necessary to move forward *under these conditions*. We also identify two extreme assumptions we are *not willing to embrace* but if true, would mean there's no selection bias...
These are: 1) D doesn’t cause M (no bias in stops). 2) No unobserved U (causes of M and Y, stops and force). U can include officer’s mood, something we’d never see in police records. We explicitly say analysts should not assume away such confounding.
But that's exactly what they did. Their key counterexample (the entirety of Section 3) is depicted below. Crucially, it assumes that U does not exist. Thus, this paper considers a different and implausible scenario that assumes away the problem.
In other words, the paper claims: if we assume, as they do below, that conditioning on a collider, M, does NOT cause selection bias, then there's no selection bias. But assuming doesn't make it true. That’s like saying "what if colliders were *not* colliders?"
They call this a "standard ignorability condition." Not even close. Standard assumptions are about ignorability conditional on *pre-treatment* variables, not *post-treatment*. And the point is trivial and absurd: assume treatment is random in your data, and you can do anything.
The whole point of our paper (below) is that we are simply unwilling to assume the problem away, unlike our critics. Instead, we ask how analysts can do something productive about it. This is not a "mathematical error," as they allege.
The claimed "mathematical error" is just out-of-context quoting from our paper. We say (below): we need X, *unless* Y. They say: X isn’t needed (error!); just *assume* Y. We know. That’s not math, and wishful assumptions != truth. We need useful ideas, not word games & gotchas.
Purposeful misreading isn't productive. To be clear, we’re not saying one cannot imagine examples where our critics' "no-selection-bias assumption" might hold. Our point: unless there’s overwhelming evidence for it, analysts must be careful.
The paper also disingenuously asserts we claim the study of racial bias is impossible given the obstacles we cite (see screenshot). This is simply false. We offer a feasible path forward in the face of these obstacles: bounds that honestly capture what we do and don’t know.
Full disclosure: we explained all this to one of the authors *in 2018* in a lengthy email exchange. They chose to write a misleading, out-of-context gotcha paper anyway. The study of policing isn’t a game. And these issues must be clarified immediately given what is at stake.
Accurately quantifying racial bias in policing is challenging, and decades of research has sought innovative ways to tackle the problem. We take our work seriously. Assuming away the challenges is reckless, and not a path we are willing to pursue.
We stand by our paper. (end)
@jgaeb1 @iamwillcai @gbasse Ravi Schroff @5harad Jennifer Hill @comppolicylab
@jgaeb1 @iamwillcai @gbasse Ravi Schroff @5harad Jennifer Hill @comppolicylab
