Thread by @drjenndowd

1/ Did the Sturgis bike rally cause 266,796 new cases of COVID-19? Probably not. Lesson- Beware viral studies that confirm your pre-existing beliefs so satisfyingly. (Long) thread: slate.com/technology/202… @FutureTenseNow @EricTopol @slate @govkristinoem @DearPandemic #Sturgis

2/ Like most people working on COVID-19, I am of the strong belief that mass gatherings during a pandemic are a bad idea. When this paper came out, the huge figures immediately hit the "I Told you So!" button in my & many people’s brains. iza.org/publications/d…

3/ The first red flag is the huge number itself-it doesn't pass the sniff test.

4/ The rally had an estimated 350-460K attendees, spread over 10 days. To reach such a huge # of 266K over 4 weeks, a LOT of transmission would have to occur both at the rally & in home counties (where cases were counted). No transmission model is mentioned, however.

5/ It requires some mental gymnastics to imagine a scenario where over 200K event attendees became infected at the rally itself. Even with bleak assumptions of 1 % of attendees already infectious (spread over 10 days), yet well enough to ride a motorcycle to South Dakota...

6/ & all of them were “superspreaders,” passing their infection along to another 10 people, back-of-the-envelope math makes it hard to get in the ballpark of this number of infections that could have happened at the rally.

7/ What about the attendees spreading the infection upon returning home? Recall the authors measured increased cases in *home* counties (defined by cell-phone pings pre-Sturgis). This was a motorcycle rally, with some "high inflow" counties far away like CA, NV & FL.

8/ Many attendees likely rode their bikes home & the lure of the open road in August after months of worldwide lockdown may have even induced many riders to take a meandering path home. (Some may have flown-I've since learned there are bike transport services for such events).

9/ Even accounting for people leaving early or going directly home, this leaves v. little time for so many infected riders to get home, infect others, incubate, get tested (with delays),& have these infections show up in county statistics by Sept. 2, just 2 weeks after the rally.

10/ How could such seemingly implausible numbers be estimated? The authors don't track individuals or contacts, but use county-level case data from home counties, & employ a "diff-in-diff" analysis--challenging under the best of times, more so when modeling epidemic spread.

11/ See the @Slate article above for more on the challenges of diff-in-diff, as well as this insightful thread(s) from @RexDouglass:

https://twitter.com/RexDouglass/status/1303379252742479872?s=20

12/ Suffice to say the paper's own figures don't inspire confidence in the assumption of parallel trends:

13/ Even taking the model at face value, the precision of the estimates suggests some caution is warranted with bold headline numbers as well:

14/ It will be hard to know exactly what went wrong with the analyses until others replicate, which I understand from @thehauer is proving a challenge:

https://twitter.com/thehauer/status/1304166170652745729?s=21

15/ .@ashishkjha also spotted that the raw data show no spikes in counties where the authors say the rally attendees came from, increasing the mystery of where the 266,796 cases could have taken place.

https://twitter.com/ashishkjha/status/1303536487259148291

16/ The authors state that the 266,796 cases, represents “19% of the 1.4 million cases in the United States between August 2nd and September 2nd.”, but their increases happened not the whole month but were concentrated in the last week, as would be expected from lag time.

17/ In this case the 1.4 million denominator isn't the best, & 266K would represent close to 45% of US cases in the last two weeks of the period, or 90% of all US cases in the last week of the period.

18/ Does this mean the rally was harmless? Definitely not. Basic transmission knowledge tells us mass gatherings and close contact are risky. Contact tracing reports have identified cases and deaths linked to the event, & this may grow over time. nbcnews.com/news/us-news/w…

19/ More broadly, I share @RexDouglass's skepticism that we can learn much from the propogation of aggregate level data analyses in pandemic times. We need individual data from contact tracing & prospective surveys to better understand transmission dynamics. Science is hard!

20/ Finally, at @DearPandemic we are committed to a critical lens regardless of whether it fits our priors. We should be *especially* skeptical of "extraordinary" claims when they fit our existing beliefs, lest we undermine trust in the science. dearpandemic.org

And thanks the power of twitter and to @Chris_Auld who helped me figure out my Meade County point was wrong-- the paper's estimate of 177 to 195 new cases in Meade county is consistent with the raw data.

15/ .@ashishkjha noticed a similar problem with the raw data for the full county analysis--no spikes in counties where the authors say the rally attendees came from, increasing the mystery of where the 266,796 cases could have come from:

https://twitter.com/ashishkjha/status/1303536487259148291?s=20

16/ The results show most increased cases in the last 1-2 weeks (especially last week), consistent w/ lag times. Still the authors state that their estimate of 266,796 cases represents "19% of the 1.4 million cases in the US between August 2nd and September 2nd."

17/ In reality, the 1.4 mill cases over the month is not the right denominator. If most Sturgis induced cases were in the last week, 266K cases represents 45% of cases 2 weeks prior or *90%* of US cases in the week prior to Sept. 2nd.

18/ Does this mean the rally was harmless? Definitely not, Basic transmission knowledge tells us mass gatherings & close contact are risky. Contact tracing reports have identified cases & deaths linked to the event & this may grow over time. nbcnews.com/news/us-news/w…

19/ More broadly, I share @RexDouglass's skepticism that we can learn much from the propagation of aggregate level data analyses during pandemic times. We need individual data from contact tracing & prospective surveys to better understand transmission dynamics. Science is hard!

20/ Finally, at @DearPandemic we are committed to a critical lens regardless of whether it fits our priors. We should be *especially* skeptical of "extraordinary" claims when they fit our existing beliefs, lest we undermine the integrity of the science. dearpandemic.org