Thread by @FitzgeraldJack_

🚨 WP alert! 🚨 I design equivalence tests for running variable (RV) manipulation in regression discontinuity (RDD), show that serious RV manipulation can't be ruled out in lots of published RDD research, and offer the lddtest command in Stata/R. 1/x

🔗: hdl.handle.net/10419/300277

Credible RDD estimation relies on the assumption that agents can’t endogenously sort their RVs to opt themselves into or out of treatment. If they can, then RDD estimates are confounded: agents who manipulate RVs are likely different in important ways from agents who don't. 2/x

Such manipulation often causes jumps in RV density at the cutoff, which can either come from genuine distributional distortions or from strategic reporting. E.g., consider the French examples below. 3/x

https://x.com/paulg/status/1799754978707575269

Good news: You can test for RV manipulation by assessing the discontinuity in the RV’s density as it crosses the cutoff. Many do so using McCrary’s (2008) DCdensity procedure. Recently, Cattaneo, Jansson, & Ma's (2018; 2020) rddensity procedure has also become popular. 4/x

Bad news: Most interpret statistically insignificant tests as evidence of negligible manipulation. This is not good practice. These tests leave researchers w/ no burden of proof to evidence their identification assumptions; absence of evidence is not evidence of absence. 5/x

My 3-step procedure can provide stat. sig. evidence that RV manipulation @ the cutoff is practically equal to zero. (1) set the largest ‘economically insignificant’ ratio ε > 1 between RV density estimates @ the cutoff. Credible ε judgments can be aggregated from survey data. 6/x

(2) Run McCrary's procedure, get logarithmic density discontinuity estimate θ. (3) Run two one-sided tests: one with Ha: θ > -ln(ε), and one with Ha: θ < ln(ε). If both are stat. sig., then that's stat. sig. evidence that RV manipulation @ the cutoff practically equals zero. 7/x

This procedure restores the burden of proof to show that RV manipulation around the cutoff is practically insignificant before ruling out meaningful RV manipulation @ the cutoff. I use it to show that RV manipulation @ the cutoff is still a serious problem for RDD research. 8/x

I leverage replication data on 36 RDD publications in top political science journals from @StommesDrew, Aronow, & Sävje () to conduct 45 RV manipulation tests. Many RVs exploited for RDD in these papers fail even lenient versions of my test. 9/xdoi.org/10.1177/205316…

@StommesDrew In this sample, > 44% of RV density discontinuities @ the cutoff can’t be significantly bounded beneath a 50% upward jump (or equivalently, a 33.3% downward jump). 10/x

@StommesDrew 50% is not a ‘special threshold’. To bring the ‘failure rate’ for my test beneath 5%, you’d have to be willing to argue that a 350% upward density jump at the cutoff is practically equal to zero. 11/x

@StommesDrew In fact, precise meta-analytic estimates suggest that for the average RV, manipulation causes an absolute density discontinuity equivalent to a 26% upward jump at the cutoff. This is likely a practically significant degree of manipulation in many relevant RDD settings. 12/x

@StommesDrew I recommend that researchers use my equivalence testing procedure to reassure against such meaningful RV manipulation around the cutoff in RDD research. In Stata, this can be done using my lddtest command. 13/x

🔗: github.com/jack-fitzgeral…

@StommesDrew In R, this can be done using the lddtest command in my eqtesting R package. 14/x

🔗: github.com/jack-fitzgeral…

@StommesDrew For further details, you can check out the working paper on my personal website. Feel free to reach out if you have any questions! 15/15

link: jack-fitzgerald.github.io/files/RDD_Equi…

@StommesDrew @threadreaderapp unroll this