Health Nerd Profile picture
Dec 7, 2021 18 tweets 5 min read
Today, our letter about a paper in Nature Scientific Reports that claimed to find no evidence that staying at home reduced Covid-19 deaths was published

This is a depressing example of how scientific error-correction fails 1/n nature.com/articles/s4159…
2/n The original paper came out in March, amid the huge worldwide epidemics, and was immediately a massive hit. After 9 months, it has been accessed nearly 400k times and has one of the highest Altmetric scores of any paper ever Image
3/n The paper has also been, I think it's fair to say, one of the more impactful pieces of work during the pandemic. It is still regularly cited everywhere to support the idea that government restrictions against Covid-19 don't work ImageImage
4/n However, there are clear issues with this paper. When it first came out, we wrote a series of threads on twitter about this as well as an OSF preprint
5/n In addition, @goescarlos wrote a detailed mathematical preprint which has now been published on the paper as well nature.com/articles/s4159…
6/n @goescarlos proved that the model the authors use will always produce what is essentially noise. @RaphaelWimmer then simulated this and proved that regardless of how closely Covid-19 deaths were linked with staying-at-home behaviour, the model finds no correlation
7/n Indeed, even if you create a dataset where staying at home increases Covid-19 deaths in a 1:1 ratio, the authors model spits out non-significant results Image
8/n There are also numerous other methodological concerns in the paper. For example, the authors uncritically use Belarussian death data, which is notoriously fake and fails the simplest of sense-checks
9/n Basically, the paper is entirely useless. It proves absolutely nothing about Covid-19 or staying at home

And yet, remember, HUGELY impactful. Used in decision-making in countries across the globe
10/n The thing is, the editors of Scientific Reports did everything right here, according to the usual academic norms. Shortly after we went public with our concerns via twitter, they posted a notice of concern on the paper Image
11/n The editors have been very responsive to us throughout. They have listened to the problems, had them peer-reviewed to make sure these are real issues, and generally done everything that we expect editors to do in this situation
12/n And yet, the entire process is a monumental failure

Why?

Because it all takes FAR TOO LONG
13/n The study went viral overnight. It was read by 100,000s within a month. While the note of concern went up quickly, it said very little and did not really disagree with the main arguments of the paper
14/n This paper, which was always useless as evidence, has been used as exactly that for nearly a year while we slowly ensured that the enormous, serious criticisms of it were accurate

That is simply not fast enough
15/n In a pandemic, decisions are made overnight. A paper which goes viral today may end up in the hands of the Albanian head of state the next day. A formal response that takes 9+ months to arrive is simply inadequate to mitigate any harms
16/n I don't have any good solutions for this. Peer-review, flawed as it is, will always allow some terrible research through. With the new online world, that research will sometimes go viral and have severe negative impacts before we can correct it
17/n I can say, however, that if it takes the better part of a year to make any major comment on a paper that mathematically cannot provide any evidence on the subject it examines, then our system for error-checking has serious issues
18/n Huge news - the paper is RETRACTED

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Health Nerd

Health Nerd Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @GidMK

Jun 20
I have no particular expertise in this area, but from a scientific perspective I find the banning of transgender/intersex athletes with these vague arguments to be problematic
The thing to remember is that no one cares whether elite athletes have an advantage. Every elite athlete has advantages over their peers to some extent (there's a reason short people rarely play basketball)

We care about UNFAIR advantages
So, for example - without debating whether this is true or not - we consider doping to be unfair for a variety of reasons while using expensive and difficult to obtain training methods are usually considered to be fair
Read 25 tweets
Jun 16
Another fairly large RCT from Iran fails to find a benefit for ivermectin in the treatment of COVID-19. People taking ivermectin no less likely to deteriorate, non-sig less likely to die, and more likely to need hospital care than placebo frontiersin.org/articles/10.33…
So that's 3 more trials with no meaningful benefits for ivermectin, with a total of about 3,000 patients, reported this week. The updated meta-analysis for mortality looks less and less positive
Another fascinating note - the new Iranian trial which found no benefit was completed nearly a year ago. The same team published positive results from their pilot in April 2021 sciencedirect.com/science/articl…
Read 6 tweets
Jun 16
It almost certainly isn't - the study found that pre/post drinking either alcoholic or non-alcoholic beer increased gut microbiome diversity by <10% as recorded by the Shannon diversity index
There were no meaningful changes to any clinical marker, and also this didn't compare to a non-beer control so there's no possible way you can infer anything about the impact of beer, specifically
I've got no idea how increasing the Shannon index of gut microbiome from 2.7 to 2.9 might relate to overall health, but technically this study didn't even show that was caused by beer per se pubs.acs.org/doi/full/10.10…
Read 4 tweets
Jun 15
Often people ask "where's the harm?" for low-risk treatments like vitamin C

A new RCT rather answers that question. In ICU, patients given IV vitamin C had a higher risk of death or major organ dysfunction than placebo nejm.org/doi/full/10.10…
This is also a fantastic teaching tool for science and evidence appraisal - the initial study indicating a possible benefit for vitamin C in the treatment of sepsis in ICU was a poorly-controlled observational trial with <100 patients
Objectively, this should not have changed treatment anywhere. But it was published in a big journal to great acclaim, and a lot of doctors took notice
Read 4 tweets
Jun 15
My favourite part about the "designed to fail" comments about new large ivermectin RCTs is that they apply to virtually all positive research conducted in 2020 as well but somehow those studies still found a benefit
"It was underdosed!"

Babalola et al used a dose of 6/12mg once every 3.5 days and had no weight-based increase. That's about 1/10-1/5 of the dose used in ACTIV-6 for a 75Kg person. Still reported a benefit Image
Mohan et al used a single dose of 12/24 mg (not per Kg). Chahla was 24mg once a week for a month. Chowdhury a single dose of .2mg/Kg. All quite a bit lower than ACTIV-6 or TOGETHER! ImageImageImage
Read 9 tweets
Jun 13
@ErinInTheMorn Another obvious error is Simpson's paradox (ecological fallacy). During the study period, somewhere around 1.5% of people aged 12-23 identified as transgender and would (in theory) be able to access GAC
@ErinInTheMorn Obviously, that's ~optimistic~, but taking it as a ballpark it means that you cannot possibly ascertain the change in rates of suicide in trans youth from an overall population rate. It's nonsensical
@ErinInTheMorn For a practical example, if suicide rates were 50/100k in trans youth vs 15/100k in cis youth, a decrease of 20% in trans youth would be entirely masked by a ~3% increase in cis youth
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(