Thread by @phl43

I'm glad the claims based on Flaxman et al.'s paper were removed, and I appreciate the willingness to quickly make corrections after I pointed out the issue, but unfortunately the new version is still misleading and should also be revised. I explain why in this short thread.

https://twitter.com/s8mb/status/1357993235088556032

I take issue with both the estimates of the number of deaths that would have been caused/averted by a later/earlier lockdown and the claim that this study addresses the criticism that people would have made voluntary behavior changes that would have slowed transmission anyway.

I start with the second claim, since once I have explain why it's mistaken, it will become clear that we can't take the estimates of the number of deaths that would have been caused/averted from that paper seriously.

This study can't possibly address the criticism that, even without a lockdown, people would have made voluntary behavior changes that would have slowed transmission, because the model on which it rests *assumes* that only NPIs or their announcement affect transmission.

Here is the description of the key part of the model in the supplementary materials. What does this mean in plain English? imperial.ac.uk/media/imperial…

Well, it means that the authors defined 12 change points during the period they studied, corresponding to public announcements, start of a NPI or partial/complete lifting of a NPI.

Each of these change point is associated to a given rate and the model assumes that the contact rate moves linearly from one to the next between each change point. The rate associated to each point is inferred by fitting the model to the data.

Thus, this study can't possibly show that people would have made voluntary behavior changes in the absence of lockdowns that would have reduced transmission, because the model literally assumes that only NPIs or their announcements affect transmission.

Of course, this also means that the study's estimates of how many deaths would have been caused/averted by starting the lockdown a week later/earlier are meaningless, because they are based on this assumption and thus also beg the question!

Indeed, in order to compute those estimates, they just fitted their model to the data (which gave them the rates associated with each change point), then just simulated a counterfactual epidemic using that fitted model where the lockdown starts later/earlier.

If the model is wrong, then the estimates of the number of deaths that would have been caused/averted are also wrong, so *at best* (there are other issues) you can only take those estimates seriously if you're already convinced that only NPIs affect transmission!

This problem is closely related to the issue I pointed out in my criticism of the initial version. The reason why, in their counterfactual, the attack rates were so ridiculously high is that Flaxman et al. also assumed that only NPI affect transmission.

https://twitter.com/phl43/status/1357737067221032960

But is this assumption wrong? Yes, it *undoubtedly* is, we have data from dozens of countries/regions — including the UK — that show it's not only wrong but *massively* wrong. So no study is ever going to show that the highlighted claim is false, because it's not.

But this is a story for another day. In this thread, I just wanted to point out that, even if you think that the assumption that only NPIs affect transmission is, if not exactly true, at least a reasonable approximation, that ICL study can't possibly show that.

Again, the study *assumes* that it's true, so it's misleading to suggest, as the new version of that page on covidfaq.co does, that it refutes the claim that people would have made voluntary behavior changes even without lockdown that would have reduced transmission.

What this study does is answer this question: if we assume that only NPIs affect transmission, and make a bunch of other largely arbitrary assumptions, what effect did each NPI had on the contact rate?

If you ask me, this has no practical relevance whatsoever, but in any case what is certain is that it doesn't in any way address the criticism that people would have made voluntary behavior changes that would have reduced transmission even in the absence of a lockdown.

The other 3 studies cited by covidfaq.co in the new version have the same or similar problems. The one I prefer is Haug et al. in Nature, but that's not to say you can take its estimates seriously, which to be clear you can't. But that's also a story for another time.

So I think that, at the very least, the page should be modified to remove the suggestion that the ICL study or indeed any other study refutes the criticism discussed above and not mention estimates of the number of deaths averted thanks to lockdowns, because they are meaningless.

P. S. I just realized that I had announced it would be a short thread and that clearly wasn't true 😅 But I swear I wasn't trying to deceive you, things just got out of hands. Anyway, I will have more to say about the problems with the case for lockdowns soonish, so stay tuned.