Thread by @FitzgeraldJack_

My paper is out in @PNASNews! I replicate a paper on the impact of COVID vaccine mandates on vaccine uptake. Removing a single bad control variable sign-flips several of the paper’s headline results. The reply’s findings are also not robust. 1/x
pnas.org/doi/10.1073/pn…

@PNASNews Rains & Richards (2024) — henceforth RR24 — reach two findings. First, RR24 claim that difference-in-differences estimates show that US state COVID vaccine mandates had imprecise impacts on COVID vaccine uptake. 2/x
pnas.org/doi/10.1073/pn…

@PNASNews Second, RR24 find that states that mandated COVID vaccination statewide now see lower uptake of COVID boosters and both adult + child flu vaccines than states that banned local COVID vaccine mandates. 3/x

@PNASNews Though RR24 imply that these differences are causal, these estimates really reflect (conditional) correlations. They’re generated by regressing uptake on a dummy variable indicating whether the state ever mandated COVID vaccines (+ some controls/interactions). 4/x

@PNASNews I was alerted to this paper by @jfeldman_epi, who raised concerns that the paper claimed to be running difference-in-differences analyses when the code shows otherwise. These concerns still appear to be valid. 5/x

https://x.com/jfeldman_epi/status/1760032277562577346

@PNASNews @jfeldman_epi My paper focuses on something different. When I was looking through the code, I noticed that all the models defending RR24’s conclusions on COVID boosters + flu vaccines controlled for contemporaneous (not baseline) COVID vaccination rates. This rang alarm bells for me. 6/x

@PNASNews @jfeldman_epi When you want to know the causal effect of COVID vaccine mandates on uptake for other vaccines, contemporaneous COVID-19 vaccination rates are a *bad control*. Controlling for them induces collider bias, which is true for two reasons. 7/x
journals.sagepub.com/doi/10.1177/00…

@PNASNews @jfeldman_epi First, both COVID-19 vaccine uptake and other kinds of vaccine uptake are driven by common unobserved factors (e.g., vaccine hesitancy). Second, COVID vaccine mandates impact COVID vaccination rates. RR24’s evidence to the contrary is quite underpowered, as they admit. 8/x

@PNASNews @jfeldman_epi Collider bias can yield misleading conclusions. Consider the ‘birth weight paradox’, where maternal smoking seems to *decrease* infant mortality after controlling for birth weight (which maternal smoking also affects). 9/x
academic.oup.com/aje/article/16…

@PNASNews @jfeldman_epi My results suggest something similar’s happening in RR24. Removing controls for COVID vaccination rates sign-flips all of RR24’s estimates on COVID booster/flu vaccine uptake. Everything goes from significantly negative to significantly positive. 10/x

@PNASNews @jfeldman_epi In their reply, RR24 respond that measures of vaccine/mandate acceptance are important controls when estimating responses to vaccination mandates, which is fair. But their empirical response is questionable. 11/x
pnas.org/doi/10.1073/pn…

@PNASNews @jfeldman_epi RR24 add controls for baseline vaccine/mandate acceptance, *but still control for COVID vaccination rates*. This is an odd choice. If you already have baseline acceptance data, then you shouldn’t risk collider bias by *also* controlling for COVID vaccination rates. 12/x

@PNASNews @jfeldman_epi Controlling for baseline vaccine acceptance closes collider paths from vaccine hesitancy. But e.g., if healthcare access affects both COVID vaccination + other vaccine uptake, controlling for COVID vaccination rates still opens collider paths (imgs: ). 13/x dagitty.net

@PNASNews @jfeldman_epi So what happens if you remove COVID vaccination rates from these new models? RR24 don’t say, and don’t offer replication data/code. However, they do point to their new data sources. I use this to extend my replication and run robustness checks on the results in their reply. 14/x

@PNASNews @jfeldman_epi Immediate issues show up in the data. E.g., RR24 say that they control for state-level CDC flu vaccination rates throughout the 2018-19 flu season. However, the CDC stores these rates monthly throughout each flu season, so there isn’t just one ‘2018-19’ rate for each state. 15/x

@PNASNews @jfeldman_epi So how are monthly vaccination rates aggregated? How are missing values handled? Are vaccinations against swine flu counted? For my part, I take a state-level mean of monthly vaccination rates against the seasonal flu over the 2018-19 flu season, ignoring missing values. 16/x

@PNASNews @jfeldman_epi I can’t exactly replicate the results in RR24’s reply, but the main estimates on the vaccination mandate dummy yield the same conclusions. So I’ve largely recovered RR24’s data and models for the additional analyses in their reply. But what about the robustness check? 17/x

@PNASNews @jfeldman_epi Again, removing the bad control sign-flips every estimate of interest in RR24’s reply. As in the original paper, the reply’s results are driven by the collider bias you get when you control for COVID vaccination rates. 18/x

@PNASNews @jfeldman_epi A more formal write-up of my response to RR24’s reply, along with an accompanying OSF repository for my response, can be found at . 19/xjack-fitzgerald.github.io/files/RR24_Res…

@PNASNews @jfeldman_epi This work has clear policy implications. Many see RR24’s findings as proof that governments and public health authorities should be wary of vaccine mandates’ ‘unintended consequences’. 20/x
cato.org/blog/unintende…

@PNASNews @jfeldman_epi But RR24 have no robust causal evidence of these ‘unintended consequences’, and more credible studies with more data show that COVID vaccine mandates significantly increased COVID vaccination in many countries. 21/x
sciencedirect.com/science/articl…

@PNASNews @jfeldman_epi My paper also provides a good example of the bad control problem in action. The bad control problem is well-established but widely unknown. This paper can be used as a teaching tool to show the problems you can run into when the bad control problem is ignored. 22/x

@PNASNews @jfeldman_epi I am grateful to @PNASNews for taking this problem seriously and publishing my replication. This entire exchange is open-access; links are provided at the next (and final) tweet. 23/24

@PNASNews @jfeldman_epi Links! 24/24

Original paper: pnas.org/doi/10.1073/pn…
Reply: pnas.org/doi/10.1073/pn…

My paper: pnas.org/doi/10.1073/pn…
Response to reply: jack-fitzgerald.github.io/files/RR24_Res…

@PNASNews @jfeldman_epi @threadreaderapp unroll this