Thread by @phl43

As promised a few weeks ago, @VC31415 and his colleagues wrote a reply to my critique of their paper. Gelman has a post on his blog with a summary of their main points. I'll probably write a reply later, but I still want to make a few points quickly 🧵 statmodeling.stat.columbia.edu/2021/10/16/cau…

If you want to read the full response, it’s available on arXiv. I have only skimmed it so far, and I don’t have time for a detailed reply at the moment, but it seems to me that I already addressed most of their points in my original post. arxiv.org/abs/2110.06136

For instance, they claim that both mask policies should have been included in the model and that when they are, the effect is significantly negative for both policies in the original data. The implication seems to be that I didn’t try a model with both policies.

First, for reasons I briefly touched upon in my post and that I might develop further when I reply, I disagree that it’s *obvious* a model that includes both policies is superior, but if it is the same argument they make here about confounding applies to their original paper!

More importantly, as I explained in my post, I *did* try models that included both policies, but even with their preferred specifications and the original data I didn’t find a significantly negative effect for both. Here is a figure that summarizes the results for one of them.

As you can see, not only are the effects not all significantly negative, but in fact only one of them is and the point estimate for another is actually positive, though not significantly so. This is a point I had already made in my post.

I don’t know what explains this discrepancy, but it seems to me that either they made a mistake or I made a mistake. And I don’t think I made a mistake... I guess we’ll see when I write a reply and take a closer look at this to try and understand the discrepancy.

Another criticism they make is that my simulation doesn’t show that their model produces biased counterfactual estimates and that I only thought otherwise because I ran the simulation for 120 days whereas they only used 96 days of data in their paper.

However, this argument is mistaken, for a reason I already explained in my post. In fact, as I noted in my post, I should arguably have run the simulation for even longer and that would have made things worse for them.

(Also, contrary to what they claim above, I never ran the counterfactual simulation for 200+ days and replaced the figure later with one for a shorter simulation, so that's not the explanation for my claim. I'll ask them what made them think that, but I have an idea.)

To be fair, they make another point in connection to my simulation that I think is both more interesting and more complicated, so I’ll address it when I reply. Nevertheless I still think my point was correct.

Another weird criticism they make is this point. I didn’t just *claim* to have done a placebo test, I *did* do a placebo test and I did easily find spurious effects. They make it sound like there can be some doubt on the matter even though I published my code!

Now, from what I understand, it seems they did a *different* placebo test and didn’t find spurious effects. My placebo test was done in the context of my simulation, whereas theirs used the real data, so the comparison doesn’t make sense.

(By the way, I just realized there was a typo in that paragraph, reduced instead of increased, so I’ll have to fix that. As I explained in footnote 73 though, it was also easy to obtain spurious negative results by randomly drawing policies from a different distribution.)

As it happens, although I did not discuss it in the post, I did perform another placebo test that used the real data and it also systematically found spurious effects, but although it was based on the real data I used a different method than them.

I think this is interesting and that trying to understand what is going on here might provide some insight, so I’ll try to replicate their placebo test and understand why, unlike my unpublished placebo test based on real data, it doesn’t find spurious effects when I reply.

Perhaps the most bizarre point they make is this criticism, according to which I don’t understand what a confidence interval is, since I claimed that since the confidence interval for business closures included zero there was no effect.

Obviously, I never made this absurd claim, here is what I actually said. This is a very different claim that doesn’t imply the basic mistake they falsely ascribe to me and, as it happens, one that is also correct.

There is more I disagree with, but again I'll write a fuller reply later and I also don’t want to make it sound everything is bad, so I will say that, while I find most of their points unconvincing and think many have were addressed in my post, this one seems interesting.

It can't explained the discrepancy between their results and mine on their first point though, since their claim in that first point was based on the original data, but I'll have to look into whether this coding error in the updated dataset I used changes things!

I’m working on a piece about the effect of population structure on transmission, which is actually directly relevant to the debate on their paper, and then I have other posts in the work, so I guess my reply to their reply will have to wait but eventually I’ll get to it.

Anyway, while I disagree with them, I still want to thank @VC31415 and his colleagues for taking the time to reply. This is definitely the best response I’ve had to any of my critiques of published work so far and I wish everyone replied in such a constructive manner!

Sure, they did call my critique "ideologically driven and overly emotional", but I compared their paper to "putting lipstick on a pig", so this is fair play in my book and I don't take it the wrong way 😂

P. S. By the way, it's kind of ironic that I'm accused of interpreting a non-significant effect as the absence of effect, since I wrote a whole post last year explaining why this was a mistake! necpluribusimpar.net/hydroxychloroq…