The story continues - after @ikashnitsky and I pointed out that this paper was mathematically impossible, and had numerous errors, it was partially corrected
His initial response to us, when we privately emailed him about the issue in the paper, was, to quote, "you are not just right because you THINK you are" (caps=italics)
Now, remember, this is not a minor paper
Altmetric of 3,400, in dozens of news articles, and it's been cited by the WHO and EU already
This paper is impacting school reopening policy across the globe
Now, it's important to note that, as far as the @JAMANetworkOpen editors and the author was concerned, that was that. We were told to submit a 600-word comment online and if the authors wanted to, they could respond
After we persisted, publishing a lengthy critique of the paper osf.io/9yqxw/ and getting a bit of press, we were invited again to submit a 600-word comment and told the authors would have to respond
And so we did. Referencing our longer critique, we fit a few of the most egregious errors into 600 words and clicked "submit" on the 7th of December
Then we waited. And waited
It was a bit surprising for us that we had to wait at all. The @JAMANetworkOpen website proudly proclaims that comments will be put up after 2 days maximum, but a month after submission...still nothing
Finally, on the 8th of January, we checked back to see that both our comment and a response from the authors had been posted
Our comment was strictly limited to 600 words (inc. references)
The authors' response? Nearly 1,500
And guess what? We were right
There was a clear mathematical error that completely reversed the results of the paper
So the authors rewrote the whole thing to try and salvage something from the ruins
Now, instead of showing that school closures kill more people than COVID-19, the paper shows that school closures either kill nobody at all or countless millions depending on whether your kids are in the US or EU
I kid you not
For some context, the new figure of YLL that is caused by school closures if you cherry-pick just the results the authors would like to use means that 3 months of closed schools has cost more YLLs than influenza in the U.S.
Since ~2015
I'm not going to go into just how awful the study is again, except to note that if you want your mind to boggle have a look at the threads below
But the real story here, to my mind, is that when junior colleagues CORRECTLY pointed out errors in a paper, they were ignored and belittled by editors and authors alike
For months
Meanwhile, the paper has been cited a dozen times academically, is in planning and policy documents across the world, despite the fact that all of this happened when the work was quite simply wrong
Now, it's very hard to guess at the impact of such a paper, positive or negative
But it is entirely possible that this study has literally killed people through the errors of the authors
Who have called us "trolls" for pointing out the mistakes
I should also note that Ilya and I get literally nothing for all of this. We have spent something like 50 hours each carefully identifying errors, writing letters, editing them down etc for no pay, mostly after work and on weekends
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I do find it fascinating how the usual "academic civility/silencing" brigade have completely ignored this whole episode. Apparently it's fine to publicly defame junior colleagues as long as it's in service of school reopening
It's also interesting to note that this is quite literally a case of academic "silencing", at least insofar as the journal and authors have tried their hardest to quash any hint of our critique
Without twitter, and the media attention it brought, I doubt very much that this paper would ever have been corrected. The only reason the authors were forced to respond is that the editors were contacted by the Guardian for comment
Thing is, it happens to all of us (I am no exception). It is commonplace to be mistaken, but rare indeed that people will own up to their errors
See, my problem is that everyone thinks that they tick most of the boxes on that list. No one thinks that they're too certain, or ignoring human health, or unkind
The entire field of epidemiology is about balancing cost and benefit, risk and reward. There is no choice without consequences, even the seemingly trivial ones
Most Master of Public Health courses (MPH) have a health economics unit for precisely this reason. Enacting a policy in one place invariably (at best) takes away resources that you would otherwise use somewhere else
This is a big part of the reason I spend so much time trying to convey nuance. There is no decision we can make for public health that is purely good
It's likely that the marginal benefit - the additional improvement on top of other things - of very restrictive COVID-19 interventions like stay-at-home orders may be quite small
However, this is probably equally true of the COST of these interventions
It's something that I've seen completely ignored by most anti-restriction campaigners, but I think it's an important point that we should consider
Yes, if you've already limited how much people can go out to restaurants etc then closing them entirely might not reduce transmission all that much
But it also won't have the same negative impact either!
3/n The paper takes 10 countries' worth of data, and compares their COVID-19 case numbers against the restrictions they had in place in early 2020, comparing those with less-restrictive non-pharmaceutical interventions (lrNPIs) with more-restrictive NPIs (mrNPIs)