Thread by @GidMK

Some movement to announce here: JAMA Open have now corrected this paper 2 months after it was published

Unfortunately, it has gone from an error-filled useless analysis to a slightly less error-filled useless analysis

Some more peer-review on twitter 1/n

https://twitter.com/GidMK/status/1327872367893176320

2/n The updated paper is here jamanetwork.com/journals/jaman…

And you can read @ikashnitsky and my original commentary on the paper here

osf.io/9yqxw/

2.5/n Important to note that this is a very influential paper. It has been in >100 news stories, and has been cited by the EU and WHO (!)

Worrying that until recently it was openly wrong

3/n In brief, the original paper made a number of obvious errors in analysis, and conducted a largely meaningless analysis using years of life lost (YLL) to compare the harms of COVID-19 and school closures

4/n So, what's changed?

Well, the main structure of the study is the same. They've taken an estimate of the impact of school closures from Argentinian children in the 70s, and applied it directly to children learning from home during the COVID-19 pandemic

5/n Previously, this estimate entirely relied on an incorrect meta-analysis that showed that every extra year of schooling reduced your risk of death by 25% over your entire lifespan

This has been removed

6/n (If I was of a skeptical bent, I'd guess that this meta-analysis was removed instead of corrected because simply correcting the model entirely reverses the main findings of the paper)

7/n Instead, the authors have completely arbitrarily divided their analysis into two sections - that based on 'US' and 'European' studies

This gives them some...interesting results

8/n If we take these results at face value, closing schools during COVID-19 in the US either killed ~10x more or 5x less people than had they stayed open, depending on whether the analysis is based on 'US' or 'European' studies

9/n So, what are these studies?

Well, here's the first obvious error that the authors have left in. The two US studies you can see in the table from the supplementaries below

10/n Problem is, Mazumder (2008) is simply a re-examination of THE SAME DATA (1919 and 1939 schooling law changes in the US) as Lleras-Muney (2005)

It's simply incorrect to just blindly chuck these estimates into the same model as if they aren't intrinsically related

11/n And it gets worse. As we pointed out in our critique, the MAIN CONCLUSION from the Mazumder (2008) paper was that SCHOOLING HAD MINIMAL OR NO IMPACT ON MORTALITY

So how did the authors instead estimate a 35.2% reduction in relative risk????

12/n It appears that the authors have cherry-picked a single result from one table of the paper that was actually just recreating Lleras-Muney's results from the original paper using the second methodological approach, rather than using the actual results from the paper

13/n It gets worse. In 2010, Mazumder published an erratum to the paper which I will quote here:

"there is little compelling evidence suggesting a causal link between education and mortality based on Census data and compulsory schooling laws"

14/n As ever, I am not here to litigate intentions, but it is amazingly strange to pick the wrong number from a table of a paper that flatly contradicts your main assertion and then lump it in with the paper ~that it was correcting~ as if this wasn't scientifically flawed

15/n I mean, using a paper that says "there is no relationship between missed schooling and mortality" to support your argument that there is a relationship between schooling and mortality is...something

16/n But this brings us back to the results. Either missed days of schooling cost 0.8 (0.1-2.4) million YLL or 13.8 (2.5-42.1) million YLL

Why the massive divergence?

17/n Well, if you use the incorrect estimates from Mazumder/Lleras-Muney, you get an average of a 45% reduction in RELATIVE RISK OF DEATH AT EVERY AGE per year of schooling

This is hilariously implausible

18/n For context, this would mean that adding 3 years of schooling for every child would reduce their risk of death to literally 0% for the rest of their lives, effectively rendering them immortal

19/n The authors justify this with some dubious language about how US studies are more similar than Nordic ones, despite the fact that the US studies are based on legal changes in 1919 and 1939, at which time the US was (for example) still segregated

20/n It is also very strange that the authors are happy to use evidence from an Argentinian study on children markedly different from those in the US but balk at studies on European children

It seems an obvious contradiction

21/n Nevertheless, what we now have is a study that, weirdly, says that either school closures have cost very few YLL or a wildly absurd overestimate, depending on whether you limit your analysis to only the studies that the authors prefer arbitrarily to use

22/n Many of the other obvious flaws in the paper remain uncorrected, and I'd urge you to read our full critique if you're interested:

osf.io/9yqxw/

23/n That being said, it is worth saying that the authors and journal have at least taken ~some~ action here, and correcting the mathematically impossible model is a good first step for this paper

24/n My hope is that now we can further correct the other obvious mistakes and issues, and come to a more realistic estimation, because this paper is currently impacting policy on an international scale

And it is still simply wrong

25/n I also think it's worth noting the process it took to get this paper corrected even this far

The original response from the lead author and journal editor was, to quote exactly, "you are not right just because you think you are"

26/n This was after @ikashnitsky and I pointed out that the paper was MATHEMATICALLY IMPOSSIBLE

Quite a worrying way to respond

27/n After we published our preprint critique, and it was reported on in the Guardian, we were told to submit a comment on the piece as soon as possible online and they'd get back to us theguardian.com/science/2020/d…

28/n Two months after the initial emails, and over a month after we submitted the comment, we have this correction published

Unfortunately, the study has already had an enormous impact, and changed lives across the world

29/n You will probably be interested @stephaniemlee @MelissaLDavey @apsmunro @DrZoeHyde @devisridhar @DFisman

30/n Overall, what we have is a paper where the mathematically impossible results have been removed, but is still flawed in numerous ways and useless as evidence for decision-making

31/n The sad fact is that the approach the authors took, if implemented correctly, probably would not have found that school closures cost more YLL than COVID-19

32/n This DOESN'T MEAN that school closures are a good thing, necessarily, but YLL is a measure inherently geared towards measuring people who have already died, and it's just not likely that closing schools has cost so much of this metric