Post-treatment selection bias can distort studies of police violence. GCBCSH proposes new stat theory for causal identification, despite selection & confounding—a huge breakthrough, if credible. We formally assess it.1/n @dean_c_knox @conjugateprior Paper: dropbox.com/s/nx8pe8gmw41d… 
Studies of racial bias in policing (e.g. Fryer 2019) often rely on detainment data (stops/arrests) to estimate discrimination in subsequent actions, e.g. use of force. If racial bias affects detainment, @sndurlauf & Heckman call this a "classic route to selection bias."
GCBSGH disagree. They say they clarify "the statistical foundations of discrimination analysis" & conclude that using their approach, "concerns about post-treatment bias may be misplaced." @jgaeb1 @iamwillcai @gbasse Ravi Schroff @5harad Jennifer Hill @comppolicylab
If GCBSGH's approach is credible, Knox, Lowe & Mummolo (KLM, 2020) is unnecessary; @sndurlauf & Heckman (forthcoming) is wrong; 40 years of research on post-treatment selection is misguided; and the design in Fryer (2019) is sound. So it’s important to get to the bottom of this.
To be clear, GCBSGH doesn’t develop new research designs or estimators. It offers an *assumption*—"subset ignorability"—to justify standard regression. It's simpler than KLM’s bounding approach, which can only show best/worst-case discrimination consistent w/ contaminated data.
Federal judges, civil rights orgs, and the general public rely on social science to answer important questions about discrimination. GCBSGH’s proposal therefore merits close investigation. What needs to be true about the world to rely on this method for credible answers?
In GCBSGH’s most general setting, we prove their method can only work if:
(post-treatment bias) = -1 * (omitted variable bias).
In other words, unless post-treatment bias happens to perfectly offset omitted variable bias, the approach will fail.
(post-treatment bias) = -1 * (omitted variable bias).
In other words, unless post-treatment bias happens to perfectly offset omitted variable bias, the approach will fail.
However, this condition is only necessary, not sufficient. Even if analysts somehow found scenarios where post-treatment bias happens to perfectly offset omitted variable bias, "subset ignorability" is not guaranteed to hold.
Even if civilians of different racial groups were as-if randomly assigned to encounters (pre-stop), this method would recover the right answer *if and only if* there’s knife-edge equality across principal strata. (In IV analyses, this is like assuming LATE = ATE.)
This isn't an as-if-random assumption. It says groups are the same *despite responding to treatment differently*. It needs minority & white stops to remain exactly balanced on the potential for officer violence, despite being stopped for very different reasons due to racial bias.
Some examples showing just how knife-edge this assumption is. To use GCBSGH’s "subset ignorability," analysts must argue (1) average strata-specific potential outcomes, weighted by (2) strata-specific probabilities, exactly balance. Not a single part of (1) or (2) is observed.
Such accidental-cancellation assumptions have been dismissed as implausible & unreliable for decades. @yudapearl once wrote it is effectively the same as "see[ing] a picture of a chair" & arguing that it may actually be "two chairs positioned such that one hides the other."
GCBSGH is motivated by a critique of KLM, which develops a cautious partial identification approach to bounding the range of possible racial bias, given these challenges. GCBSGH reject all of KLM because, they claim, it misuses the word "necessary" when describing assumptions.
We clarify lingering confusion on terms. In short, our assumptions are necessary *for our goals*. We also show every GCBSGH counterexample alleged to invalidate KLM was already discussed therein, but dismissed due to implausibility.
Another clarification: despite prominent claims to the contrary, KLM also proves selection bias contaminates both total effects (ATE, ATT) & controlled direct effects (CDE, the effect of race after a police stop). Changing estimands doesn’t solve the problem. (cc @StatModeling)
GCBSGH formalize an assumption that is often left implicit. This is a valuable contribution, and allows for a precise debate on the merits of their approach. We have learned a lot from investigating these issues more closely. All new methods deserve scrutiny, including ours.
However, we stand by our original paper. Post-treatment selection threatens the study of racial bias in policing. GCBSGH’s proposed solution rests on assumed accidental cancellation that can’t be verified or credibly relied upon in this setting, even in gold-standard experiments.
Post-treatment selection has been shown to threaten inferences in a host of settings, even in experiments. We see no reason the policing context would be exempt from this concern, especially given the complex multi-stage process that raises unusual threats to causal inference.
In light of these obstacles, we advocate cautious and rigorous approaches to bounding effects, or design-based solutions that mitigate the threat of selection from the start. This topic demands the highest scientific standards. (end)
