Thread by @phl43

Okay, I know most people aren't going to read this because it's long (though you should!), so let me go over the main points quickly. 1/n

https://twitter.com/phl43/status/1334892503930187777

It's a critique of Flaxman et al.'s paper, which is constantly cited as proof that lockdowns are the only interventions that really works. It's already been cited almost 450 times even though it was only published in June. 2/n nature.com/articles/s4158…

It found that only lockdowns really had a meaningful effect on transmission in Europe during the first wave. Here is a chart that show the results in a few countries that were included in the study. 3/n

As you can see, they find that lockdowns massively and suddenly cut transmission, while other interventions had almost no effect. However, the fact that R has this shape (a stepwise function) is baked into the model, so it was bound to find something like that. 4/n

When they compare their model to a counterfactual in which there were no interventions, they find that interventions saved more than 3 million people, and since only lockdowns had a meaningful effect, this was mostly thanks to lockdowns. 5/n

So it sounds like lockdowns are pretty damn awesome! However, as I explain in the post, something is weird about the results. Could it be that lockdowns aren't *that* awesome after all? 🤔 6/n

What's weird about the results is that, while they find that only lockdowns have any effect on transmission, the results look pretty similar in Sweden than in the rest of Europe. Yet there was no lockdown in Sweden! So how is that possible? 7/n

When I noticed that, my bullshit detector went on full alert mode. Something had to be wrong, but there was no explanation in the paper. So I downloaded the code reproduced the results and decided to take a closer look at them 🧐 8/n

I won't go into the nasty details here, read the post, but they included a country-specific effect in the model. It's supposed to model the fact that even the same interventions aren't going to have exactly the same effect in different countries for idiosyncratic reasons. 9/n

For instance, perhaps Germans are more rule-abiding, so the lockdown was somewhat more effective over there. Or maybe broadband in Spain sucks, so remote work was more difficult, which reduced the effectiveness of the lockdown. That sort of things. 10/n

The only way I could make sense of the results, i. e. only lockdowns work but Sweden somehow reduced transmission about as much as other countries despite not having a lockdown, was that this country-specific effect was huge for Sweden. 11/n

So I ran the code and plotted the country-specific effects and, surprise, surprise, my hypothesis was exactly right. The country-specific in Sweden is *gigantic*. So much so that the probability of an effect that large was only 1 in 4,000 according to their prior... 12/n

Now, unless you believe there are magical fairies in Sweden that made banning public events over there several orders of magnitude more effective than anywhere else, this means that something is wrong with the model. 13/n

Another obvious problem when you plot the effects is how insanely wide the credible intervals are. Obviously there is something wrong with the model, so you can't trust any of the conclusions of this paper! 14/n

But amazingly none of that made it into the paper! They do all sorts of bullshit robustness analyses in the supplementary material, but nowhere in the paper or the supplementary material do they mention this small detail about the country-specific effect in Sweden 🤔 15/n

But this small detail totally undermines the conclusion of their paper, which is probably why they swept it under the rug, because otherwise I doubt their paper would have been published in Nature. Ladies and gentlemen, I present to you, Science! 16/n

While I do not doubt that lockdowns reduced transmission, it's obvious that less stringent interventions were enough to push R below 1. You literally just have to eyeball a chart that shows the deaths curve in Sweden, where there was no lockdown, to see that! 17/n

What Flaxman et al. did is just use sophisticated methods to reach a conclusion that was obviously false. But since they used all sorts of scary-looking mathematical formulas, people forget about the assumptions and just assume that the conclusion must be right. Science! 18/n

To estimate that lockdowns saved more than 3 millions people, Flaxman et al. assumed that, without interventions, nothing would have happened. People's behavior wouldn't even have changed, so only immunity in the population would eventually have stopped the epidemic. 19/n

This is obviously totally unrealistic, but it also doesn't tell us how many people would have died if there had been interventions but no lockdown, which is arguably a more interesting question. 20/n

However, you'll have to wait a little for me to answer this question, because we're having drinks over Zoom with my friends and they're waiting for me. I'll be back when we're done! 21/n

Okay, I'm even drunker than when I wrote the first part of this thread, but I'm back. By the way, we didn't actually use Zoom, we used Jitsi. But I have to a reputation as a sellout to China to uphold, so I pretended that we were using Zoom. 22/n

Anyway, like I said above, Flaxman et al.'s counterfactual is not very interesting, because nobody has ever argued that we shouldn't do *anything* to contain the epidemic. 23/n

The question is whether* lockdowns* were really necessary to prevent millions of deaths or could less stringent interventions, while resulting in more deaths, have done so too. As I noted, we already know it's the latter, because Sweden proved that. 24/n

Now, even though this is clearly true, it doesn't imply that lockdowns weren't the best option from a cost-benefit perspective. It also doesn't imply that lockdowns weren't the best option given the information we had in March. 25/n

Those are different questions, which I do not address in my post. For what it's worth, with the benefit of hindsight I think lockdowns clearly weren't the best option from a cost-benefit perspective, but I'm not sure about the other question. 26/n

The question I want to ask here is, if we take Flaxman et al.'s results at face value (which I argued you shouldn't), how many lives were saved by lockdowns? As I already noted, we can't answer this question with their counterfactual. So I used a different counterfactual. 27/n

In this counterfactual, instead of doing nothing to contain the epidemic, every country is adopting the same policy as Sweden and I'm assuming it has the same effect on transmission as it had in Sweden according to Flaxman et al.'s model. 28/n

Here is a table that shows how many lives were saved if you compare what actually happened to what would have happened in this alternative counterfactual. I call it "from 3.1 million lives saved to 200,000 real fast" 😏 29/n

In the blog post, I use another version of the model that is estimated on each country separately to do the same thing (instead of the partial pooling + country fixed effect they used in the paper), and the difference is even more striking. 30/n

But it doesn't really matter, I don't think you should take those estimates very seriously either since they're still based on Flaxman et al.'s obviously misspecified model, and the little trick I used doesn't really provide us with a good counterfactual either. 31/n

The point is that, while Flaxman et al. conclude that only complete lockdowns have a meaningful effect on transmission, it's clear when you use *their own estimate* of how much Sweden's no-lockdown policy that less stringent interventions were enough to do most of the work. 32/n

There is more to say about this, and that's why you should read the whole blog post, but like I said I'm a bit drunk at the moment and I think this thread covers the most important points I wanted to make. Please read the post if you're interested in the details. 33/33

P. S. By the way, the figure for Sweden is wrong in the table I use in the post about this, though it shouldn't affect the other countries. I'll post the corrected version later, but I need to run the algorithm again and it takes forever on my laptop.

https://twitter.com/phl43/status/1335025513027547136