Thread by @GidMK

Ok, so I've had a read of this paper (which has been all over the news) that says quite explicitly that closing schools will probably cause more years of life lost than leaving them open in a pandemic

Let's do some peer-review on twitter! 1/n

https://twitter.com/GidMK/status/1327791987014074369

2/n Paper is here. It's actually very simple, basically the authors took an estimate of how many days of school kids lost due to closures during COVID-19, an estimate of how much that impacts their years of life, and multiplied out

jamanetwork.com/journals/jaman…

3/n Specifically, they said:

1. Children miss days of school
2. Days of school cause decreased educational attainment
3. Decreased educational attainment causes less life

4/n They then compared this with estimates of the number of years of life lost due to COVID-19 already, and found that the loss of educational attainment caused more death (with 98.1% probability!)

5/n Now, that sounds very robust. 98.1% probability is extremely certain!

It's also, as far as I can tell, an incredibly meaningless number. NONE OF THIS IS THE SLIGHTEST BIT CERTAIN

6/n Firstly, the paper that all of this extrapolation is based on - reference 14. This is where the authors got the relationship between days of schooling missed and overall educational attainment

7/n This is a long and fascinating examination of the association between teachers' strikes in Argentina and the eventual earnings that children achieved when they grew up

8/n Note there that I said ASSOCIATION. This is a key point. The authors of this paper WARN AGAINST USING IT CAUSALLY because, among other issues, the data for children in Argentina in the 70s and 80s (especially schoolkids) was not great

9/n Indeed, the main analysis of this paper explicitly excludes the impact of missing secondary education because their data was incomplete and many Argentinian children in the cohort at the time did not attend secondary school

10/n It should also be noted that the impact of TEACHERS' STRIKES on ARGENTINIAN CHILDREN may not mean much when we're talking about American children, some % of whom could learn at home and thus were not explicitly missing class

10.5/n (Also, whether the impact of teachers formally striking on children in the 70s is the same as missing days of in-person education during a pandemic in 2020...I would say probably not)

11/n Next, the authors did something odd. They decided to compare this to the DIRECT mortality from COVID-19. This means that they are only counting people who have died as a direct result of the disease

12/n But, if we're comparing the total impact of missed education on children over a lifetime, this makes no sense. COVID-19 has many other harms, after all, some of which are disputed but many of which are easily attributable

13/n Where are the years of lost life due to ICU stays? Hospitalizations? Disability, mental health issues, avoided care due to full hospitals, etc etc etc???

14/n (This is also a problem for the YLL calculation for children missing school, FYI, it is FAR more complex than just missing educational attainment. So many potential things that are not included here!)

15/n Ultimately, what we get in the end - the "98.1% probability" figure - makes almost no sense at all. Even if the Argentinian study was applicable to the US situation, which it clearly isn't, the counterfactual is just inappropriate

16/n It may be possible to do a rigorous cost/benefit of school closures, but I really don't see how this can be considered one

17/n At best, what we've got here is a very rough inference about what the impact of school closures could possibly have been (as long as US kids in 2020 are the same as Argentinians in 1977), compared to an undercount of the impact of what COVID-19 probably was

18/n I should add - I am not saying that closing schools is necessarily a positive in this thread. What I AM saying, however, is that we could not possibly know whether it was or not from this paper

19/n Many further criticisms of this paper that I missed in this also excellent thread. It appears that almost none of the assumptions in the paper make any sense at all

https://twitter.com/ikashnitsky/status/1328121853970436100?s=20

20/n Another point I've been thinking about - the authors fail to consider the impact of an ongoing epidemic on school attendance. Lots of cases in the local area probably impacts schooling as well!

21/n Also, what's the impact of parents/carers getting sick and dying? Presumably this will have an outsized impact on the children it affects, which would cause (based on this model) large increases in YLL

22/n Ok, there's another HUGE problem with the study

The findings are entirely based on the idea that an extra year of schooling reduces your risk of death by 25%, a figure reached by aggregating studies in a meta-analysis model

23/n This number is the ENTIRE BASIS for the paper. If schooling reduces mortality by less than 25%, then the number of deaths caused by missed school days will drop as well

And we have a problem, because this 25% figure is...wrong

24/n Firstly, we have these top 2 papers. Because they're from the US, they are more heavily weighted than the others (this is arbitrary, but whatever)

Except the values for the second paper are flatly incorrect

25/n Mazumder (2008) did not find a relative risk of 0.65 (0-1.3), it found an RR of 0.89 (0.74-1.04)

The paper is here. I've read it very carefully and this appears to be the main finding from the author chicagofed.org/publications/e…

26/n So the number is wrong, but this brings us to another issue - the two papers at the top that were double-weighted...are ON THE SAME DATASET

This is a massive issue

27/n You see, Lleras-Muney compiled an estimate based on a number of factors on the reduction in risk from 1 year of extra schooling. Mazumder, three years later, took that same database and re-analyzed it considering additional factors, and estimated a lower reduction

28/n However you slice it, it's inappropriate to just bung these two estimates together into a model and treat them as separate. I would argue that the Mazumder paper is probably a better estimate, but either way what they've done here is wrong

29/n If I re-run the meta-analysis they've purported to use with the correct numbers, we end up with this graph instead. A relative reduction in risk of death of 5% for every year of schooling

This is A QUARTER of the estimate used in the study

30/n So AT BEST assuming there are no other errors, the estimate of the years of life lost due to school closures should be divided by 5