Thread by @GidMK

Recently this paper was published purporting to show that staying at home does not prevent COVID-19 deaths

I don't think the evidence provided supports that at all!

Some peer-review on twitter 1/n

https://twitter.com/jhnhellstrom/status/1368585541462208519

2/n Paper is here, it's a pretty simple ecological study comparing countries on their deaths/million from COVID-19 and Google mobility data nature.com/articles/s4159…

3/n The authors modelled the impact of time spent in "residential" areas as shown by Google against number of COVID-19 deaths in different areas, and in most cases found that there was no significant explanatory power for this model

4/n In other words, if you compare places that had people spending more time in "residential" areas against those that didn't, they had similar COVID-19 deaths per million

5/n The authors ensured that the countries/regions were reasonably comparable by controlling for a few population measures like markers of income and healthcare

6/n Now, the first issue is a pretty obvious one that springs out immediately:

Google "residential" mobility data

7/n Firstly, this is a selected dataset. Only people who use Google services (mostly Android users) AND HAVE LOCATION HISTORY TURNED ON are represented in this dataset

Almost certainly not representative of the people who are mostly dying from COVID-19

8/n This is mentioned in a sentence in the discussion, but I think it's a fundamental issue that makes this analysis a bit useless. We know that 50%+ of COVID-19 deaths are in the over-65 population, who are the least likely to be represented in this dataset!

9/n Furthermore, only using the "residential" data*, as the authors did, is a big problem

You see, most people already spend most of their time at home

*there's also an issue with how opaque the term "residential" is and how this is calculated, but one issue at a time

10/n Google even points this out in the explainer for mobility data. Most people already spend 12+ of their 24 hours a day at home, so the "residential" category is the LEAST LIKELY to show any increase/decreases

11/n It makes sense when you remember that Google mobility data tracks CHANGES, not absolute figures. So 50% of the population working from home 100% would reduce office mobility by 50%, but only increase residential by a fraction of that amount

12/n For example, here is the "residential" vs "workplace" mobility data for the state of Victoria in Australia during their mammoth lockdown. "Residential" never goes above a 25% increase, but "workplace" decreases FAR more

13/n What this means is that by comparing "residential" mobility, you are the most likely to find no difference by default. This is called a bias towards the null, and it's not ideal

14/n Furthermore, remember my asterisk from above? Yeh, turns out that it's really hard to find out what "residential" actually means, how it's calculated, or what the raw figures are based on, presumably because this is proprietary Google analysis

15/n So the conclusions about staying at home make no sense at all. "Residential" mobility data might not have been different between places, but for all we know that's a meaningless measure anyway that has very little to do with how much people stay at home

16/n On top of that, this study suffers from the same drawbacks that most ecological trials do. To their credit, the authors acknowledge this in the discussion, but it certainly hasn't filtered through to the public

17/n Limitations inherent in this sort of research are many and varied, but as one example it's hard to make any realistic inferences about individuals staying at home when your unit of study is Spain vs the United States of America

18/n Even within Australia, which was included, the massive Victorian outbreak/lockdown skew the figures enormously, because one state with 1/4 of the population locked down while the rest of the country opened up

19/n We might actually expect null findings from an ecological trial of this sort, because at the country level heterogeneity in local policy irons out a lot of the impact

20/n It's also worth noting that the study literally does not address the question of whether government orders influenced COVID-19 deaths. Even if you ignore all the other issues, "residential" mobility data simply can't answer that question!

21/n There are many reasons that people stay at home, and given the opacity of "residential" data it's hard to say much about the results other than that this is a hard question that we may never answer well

22/n That being said, the idea that this study disproves staying at home as a driver of COVID-19 mortality is obviously wrong - at best, it is an example of how difficult answering that question can be