This paper has been doing the rounds, claiming that lockdown was useless (the source of the 0.2% effect of lockdown claim). Dozens of people have asked my opinion of it, so here we go:
In my opinion, it is a very weak review that doesn't really show much, if anything 1/n
In my opinion, it is a very weak review that doesn't really show much, if anything 1/n
2/n The paper is a systematic review performed by three very highly-regarded economists who have also been extremely anti-lockdown since March 2020. You can find it here: sites.krieger.jhu.edu/iae/files/2022…
3/n As others have noted, this is a "working paper", which essentially means it's not peer-reviewed and reflects only the opinions of the three authors named
4/n The first issues with the review itself are very basic - the inclusions/exclusions
None of the criteria make much sense to me
None of the criteria make much sense to me
5/n For example, all papers with modelled counterfactuals are excluded. Because this is the most common method used in infectious disease assessments, this has the practical impact of excluding most epidemiological research from the review
6/n The authors claim that they only include studies using a "difference in difference approach", but according to other economists, well...
https://twitter.com/AndreasShrugged/status/1488993038915489794?s=20&t=UyUq2tuM-dCCT45ysF_HJA
7/n So, the included studies certainly aren't representative of research as a whole on lockdowns - not even close. Many of the most robust papers on the impact of lockdowns are, by definition, excluded
8/n We can see the impact of this in the main table of results. This is the source of that 0.2% figure you might've seen thrown around
This is just a useless meta-analysis
This is just a useless meta-analysis
9/n Why is it useless?
Well, let's look at the studies here
Bjørnskov = Oxford stringency index correlated to crude death reports
Shiva/Molana = Oxford stringency index correlated to crude death reports
Stockenhuber = well, you get the gist
Well, let's look at the studies here
Bjørnskov = Oxford stringency index correlated to crude death reports
Shiva/Molana = Oxford stringency index correlated to crude death reports
Stockenhuber = well, you get the gist
10/n The only differences in these studies are the countries included, timing, corrections for confounding, and lags. Also some modelling differences
11/n One study that's noticeably absent is the Hale et al paper which PRODUCED THE OXFORD STRINGENCY INDEX ON WHICH ALL OF THIS IS BASED
They estimated a massive reduction in death due to lockdown
They estimated a massive reduction in death due to lockdown
12/n If you've got 7 papers that take the same databases and manipulate them in different ways, it doesn't really make sense to calculate a mean weighted by standard error and call that the result. It's just bizarre
13/n But it gets even weirder. If you look at the model, almost the entire weighting is based on this paper, Chisadza et al
But Chisadza et al found a BENEFIT for lockdowns
But Chisadza et al found a BENEFIT for lockdowns
14/n Indeed, the authors of this paper have publicly disagreed with the review, and accused the review authors of having a predetermined conclusion when writing the paper
https://twitter.com/whippletom/status/1489229847973015565?s=20&t=UyUq2tuM-dCCT45ysF_HJA
15/n If you look closely, it seems that there are similar issues with quite a lot of the included research. For example, in table 5 much of the aggregate model is based on this paper...
16/n ...the senior author of which has agreed that you can't really interpret their regression estimates in this way
https://twitter.com/GidMK/status/1489386417885040643?s=20&t=UyUq2tuM-dCCT45ysF_HJA
17/n Another included paper found that significant restrictions were effective, but is included in this review as estimating a 13.1% INCREASE in fatalities. The maths used to derive this is pretty opaque
18/n All of this adds up to a very weird review paper. The authors exclude many of the most rigorous studies, including those that are the entire basis for their meta-analysis in the first place
19/n They then take a number of papers, most of which found that restrictive NPIs had a benefit on mortality, and derive some mathematical estimate from the regression coefficients indicating less benefit than the papers suggest
20/n There's no real rating of the potential for bias in the included research (the review only uses these 4 really bizarre 'quality dimensions' which...well)
21/n All of this together means that the actual numbers produced in the review are largely uninterpretable. The "quality" assessment is meaningless, and the numbers themselves not really correct
22/n In reality, the impact of "lockdowns" is very hard to assess, if for no other reason than we have no good definition of "lockdown" in the first place. The authors of this review define a "lockdown" as LITERALLY ANY INTERVENTION
23/n However, this is certainly not how the analysis is actually done. In most cases, it seems the authors have taken estimates for stay-at-home orders as their practical definition of "lockdown" (this is pretty common)
24/n And honestly, I'd agree that the evidence for marginal benefit from stay-at-home orders once you've already implemented dozens of restrictions is probably quite weak. One of the included studies, Spiegel et al, makes this point rather well academic.oup.com/rfs/article/34…
25/n That being said, if we consider "lockdown" to be any compulsory restriction at all, the reality is that virtually all research shows a (short-term) mortality benefit from at least some restrictions
26/n You can even see this IN THE REVIEW. The authors found benefits for compulsory facemasks, business closures, border closures/quarantine, and school closures. Some of the benefits were very large!
27/n Imo if we define "lockdown" as "the marginal benefit of stay-at-home orders on top of many other restrictions", it's probably fair to argue from this paper that the benefit might be quite small
28/n Indeed, that's been shown before, and is quite a reasonable position based on the evidence! The -0.2% figure is pretty meaningless, but the general idea is not totally wild
29/n If we define "lockdown" as "any compulsory restriction against COVID-19", however, this paper actually shows quite the opposite, that many compulsory restrictions are very effective at controlling the disease
30/n All of this comes with the huge proviso that the estimates in the paper probably aren't very useful, and the specific numbers are likely to be quite a bit off
31/n As a minor addendum, I would also note that I personally agree that a lot of people originally underestimated the impact of voluntary behaviour change on COVID-19 death rates - it's probably not wrong to argue that lockdowns weren't as effective as we initially thought
32/n You can actually see this quite fascinatingly in new research by @JanMBrauner and co, which showed that the impacts of restrictions in the second wave was decidedly different to the first nature.com/articles/s4146…
33/n HOWEVER, this runs both ways - it is also quite likely that lockdowns did not have the NEGATIVE impact most people propose, because some behaviour changes were voluntary!
34/ Some more excellent critique of the very bad 0.2% estimate
https://twitter.com/ProfPHansen/status/1489389713777696769?t=ScruHd3uoCBCBrgfoiMY3w&s=19
35/n Worth noting there are numerous other issues with this study, but at 34 tweets this is getting far too long. We will be publishing an academic commentary shortly, but suffice to say it's an extremely poor-quality study that is not useful as evidence for anything
• • •
Missing some Tweet in this thread? You can try to
force a refresh
