Health Nerd Profile picture
11 Jan, 33 tweets, 10 min read
Some movement to announce here: JAMA Open have now corrected this paper 2 months after it was published

Unfortunately, it has gone from an error-filled useless analysis to a slightly less error-filled useless analysis

Some more peer-review on twitter 1/n
2/n The updated paper is here jamanetwork.com/journals/jaman…

And you can read @ikashnitsky and my original commentary on the paper here

osf.io/9yqxw/
2.5/n Important to note that this is a very influential paper. It has been in >100 news stories, and has been cited by the EU and WHO (!)

Worrying that until recently it was openly wrong
3/n In brief, the original paper made a number of obvious errors in analysis, and conducted a largely meaningless analysis using years of life lost (YLL) to compare the harms of COVID-19 and school closures
4/n So, what's changed?

Well, the main structure of the study is the same. They've taken an estimate of the impact of school closures from Argentinian children in the 70s, and applied it directly to children learning from home during the COVID-19 pandemic
5/n Previously, this estimate entirely relied on an incorrect meta-analysis that showed that every extra year of schooling reduced your risk of death by 25% over your entire lifespan

This has been removed
6/n (If I was of a skeptical bent, I'd guess that this meta-analysis was removed instead of corrected because simply correcting the model entirely reverses the main findings of the paper)
7/n Instead, the authors have completely arbitrarily divided their analysis into two sections - that based on 'US' and 'European' studies

This gives them some...interesting results
8/n If we take these results at face value, closing schools during COVID-19 in the US either killed ~10x more or 5x less people than had they stayed open, depending on whether the analysis is based on 'US' or 'European' studies
9/n So, what are these studies?

Well, here's the first obvious error that the authors have left in. The two US studies you can see in the table from the supplementaries below
10/n Problem is, Mazumder (2008) is simply a re-examination of THE SAME DATA (1919 and 1939 schooling law changes in the US) as Lleras-Muney (2005)

It's simply incorrect to just blindly chuck these estimates into the same model as if they aren't intrinsically related
11/n And it gets worse. As we pointed out in our critique, the MAIN CONCLUSION from the Mazumder (2008) paper was that SCHOOLING HAD MINIMAL OR NO IMPACT ON MORTALITY

So how did the authors instead estimate a 35.2% reduction in relative risk????
12/n It appears that the authors have cherry-picked a single result from one table of the paper that was actually just recreating Lleras-Muney's results from the original paper using the second methodological approach, rather than using the actual results from the paper
13/n It gets worse. In 2010, Mazumder published an erratum to the paper which I will quote here:

"there is little compelling evidence suggesting a causal link between education and mortality based on Census data and compulsory schooling laws"
14/n As ever, I am not here to litigate intentions, but it is amazingly strange to pick the wrong number from a table of a paper that flatly contradicts your main assertion and then lump it in with the paper ~that it was correcting~ as if this wasn't scientifically flawed
15/n I mean, using a paper that says "there is no relationship between missed schooling and mortality" to support your argument that there is a relationship between schooling and mortality is...something
16/n But this brings us back to the results. Either missed days of schooling cost 0.8 (0.1-2.4) million YLL or 13.8 (2.5-42.1) million YLL

Why the massive divergence?
17/n Well, if you use the incorrect estimates from Mazumder/Lleras-Muney, you get an average of a 45% reduction in RELATIVE RISK OF DEATH AT EVERY AGE per year of schooling

This is hilariously implausible
18/n For context, this would mean that adding 3 years of schooling for every child would reduce their risk of death to literally 0% for the rest of their lives, effectively rendering them immortal
19/n The authors justify this with some dubious language about how US studies are more similar than Nordic ones, despite the fact that the US studies are based on legal changes in 1919 and 1939, at which time the US was (for example) still segregated
20/n It is also very strange that the authors are happy to use evidence from an Argentinian study on children markedly different from those in the US but balk at studies on European children

It seems an obvious contradiction
21/n Nevertheless, what we now have is a study that, weirdly, says that either school closures have cost very few YLL or a wildly absurd overestimate, depending on whether you limit your analysis to only the studies that the authors prefer arbitrarily to use
22/n Many of the other obvious flaws in the paper remain uncorrected, and I'd urge you to read our full critique if you're interested:

osf.io/9yqxw/
23/n That being said, it is worth saying that the authors and journal have at least taken ~some~ action here, and correcting the mathematically impossible model is a good first step for this paper
24/n My hope is that now we can further correct the other obvious mistakes and issues, and come to a more realistic estimation, because this paper is currently impacting policy on an international scale

And it is still simply wrong
25/n I also think it's worth noting the process it took to get this paper corrected even this far

The original response from the lead author and journal editor was, to quote exactly, "you are not right just because you think you are"
26/n This was after @ikashnitsky and I pointed out that the paper was MATHEMATICALLY IMPOSSIBLE

Quite a worrying way to respond
27/n After we published our preprint critique, and it was reported on in the Guardian, we were told to submit a comment on the piece as soon as possible online and they'd get back to us theguardian.com/science/2020/d…
28/n Two months after the initial emails, and over a month after we submitted the comment, we have this correction published

Unfortunately, the study has already had an enormous impact, and changed lives across the world
30/n Overall, what we have is a paper where the mathematically impossible results have been removed, but is still flawed in numerous ways and useless as evidence for decision-making
31/n The sad fact is that the approach the authors took, if implemented correctly, probably would not have found that school closures cost more YLL than COVID-19
32/n This DOESN'T MEAN that school closures are a good thing, necessarily, but YLL is a measure inherently geared towards measuring people who have already died, and it's just not likely that closing schools has cost so much of this metric

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Health Nerd

Health Nerd Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @GidMK

13 Jan
A new paper has been published by John Ioannidis and Jay "Great Barrington Declaration" Bhattacharya on "lockdowns" as a COVID-19 preventative measure

Let's do some twitter peer-review! 1/n
2/n The paper is here, and it's an interesting read:

onlinelibrary.wiley.com/doi/epdf/10.11…
3/n The paper takes 10 countries' worth of data, and compares their COVID-19 case numbers against the restrictions they had in place in early 2020, comparing those with less-restrictive non-pharmaceutical interventions (lrNPIs) with more-restrictive NPIs (mrNPIs) Image
Read 25 tweets
11 Jan
One thing that's quite funny to note about the awful @JAMANetworkOpen study that has recently been incompletely corrected is that it implies that school holidays are killing people
See, the authors assume that, in the US, every additional year of schooling reduces your risk of death by ~46% across the entire lifespan, and that any/all days missed in terms of school are precisely equivalent to missing schooling
This is not some vague sidepoint, but a central assumption underlying the entire model. Every day missed from school is precisely equivalent to missing lifetime schooling by a fixed amount per child Image
Read 8 tweets
11 Jan
I do find it quite remarkable that people who have been making testable predictions that have completely failed to come through every day for MONTHS are still being given so much air
For example, a testable prediction made by Sunetra Gupta, Anders Tegnell, and others was that areas most impacted by COVID-19 in March/April would be substantially protected from any resurgence. This has proven largely wrong
This was, in part, based on the prediction by Gupta that the UK (and others) had already reached "herd immunity", or were close to it

Also wrong
Read 4 tweets
10 Jan
GETTING SICK IS NOT A MORAL FAILING
Yes, this applies to COVID-19 as well. Stop blaming people for being sick
"But they didn't wear a mask" lots of people who DID wear a mask got COVID-19, it's not perfect protection, you can't apply morality to something that is largely out of your control
Read 4 tweets
10 Jan
"Lockdowns don't work, you can't control COVID-19 with govt restrictions!"

Pretty much every state in Australia has now controlled an exponential outbreak using a variety of restrictions, if we can do it you can too
Hopefully we can get enough people vaccinated by mid-2021 to help alleviate the issue anyway, but it remains remarkably ignorant to say that govt action can't control COVID-19 outbreaks when it very clearly can
Whether you WANT to control the disease with such measures is a very different question, but the obvious fact that you CAN is really not up for debate
Read 4 tweets
3 Jan
Statements made by @MichaelYeadon3 by date, against the number of mechanically ventilated COVID-19 patients in the UK

At a certain point, I do wonder why people still listen
And yes, for anyone interested in facts, most positive SARS-CoV-2 tests are true positives (even in the UK), this is not some conspiracy but actually quite simple mathematics

gidmk.medium.com/most-positive-…
"This is just like previous years' influenza"

In 2017/18 and 18/19 the peak admissions across the UK to ICU/HDU units for flu did not top 300 per week, making the current situation at least 5x worse than your average season

And it's getting worse
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!