2/n The study is here, and it's a cluster-randomized controlled trial, where people living in dorms of Singapore were given one of the 4 treatments or a vitamin C control during large COVID-19 outbreaks in the dorms ijidonline.com/article/S1201-…
3/n The results seem to show that people who take HCQ or P-I have fewer infections than those who only have vitamin C, with a really impressive risk reduction
4/n If these reductions are true, either HCQ or P-I would be a massive, world-changing reduction in risk for the prevention of COVID-19 (although zinc and ivermectin would be largely useless)
5/n But reading the methodology, some things immediately spring out. Firstly, this was a cluster-randomized trial, where participants were randomized to the intervention by the floor of their dorm
6/n ...but this was NOT what the study was originally designed for. The sample sizes, statistical analysis, and other protocols were designed for a parallel-arm clinical trial
This is not ideal
7/ While cluster-randomized trials are not my forte, all of the statisticians I've asked so far have said that turning a 5-arm study into a cluster RCT in this way leaves you with quite a lot of issues
7.5/n In particular, by clustering at the floor level the researchers essentially reduced their effective sample size from 3,000 patients to 40 floors. You have to take this into account in the statistical analysis, which the study does not seem to have done
8/n There's also an issue with how COVID-19 was ascertained in the study. Patients gave a blood sample at day 1 and 42, and then medical records were searched to see if they'd had a positive PCR between those two dates
9/n So, firstly, that is an issue because there's a bias in the PCR testing - only those who had a test (for whatever reason) AND had it entered into whichever medical records system would show up as positive
10/n On top of that, the serology isn't great. Serology often misses people who are tested early in their infection, so the estimates of infection rates for all the groups are probably too low
11/n If you correct for this, instead of the percentages given in the results, you get much higher values for all the treatments, and a potentially reduced protective effect
12/n Also, the study excluded people who already had COVID-19 antibodies at the start, but there isn't much information about these people, which is an issue given that they were excluded after already being randomized (and treated?)
13/n Moreover, a full 20% of the sample was excluded because they were not from randomized floors, which is bizarre. How were they recruited, randomized, and apparently treated if they were not on randomized floors?
14/n It seems very much like this trial was conceived, planned, AND RUN as a randomized parallel-arm trial and then halfway through switched to a cluster RCT
15/n Ok, so that's all pretty worrying. The results may not be statistically viable, the methodology has quite a few flaws
But there's actually a potentially bigger issue
16/n The primary outcome reported in the trial was laboratory-confirmed (through either PCR or serology) COVID-19. This is the measure that the headline results is based on
17/n But if we go to the pre-registration for the study (which, incidentally, doesn't talk about controlling for a clustered design), there's something a bit weird
The primary outcome was originally "acute respiratory illness"
18/n So when the study was registered, in June 2020, the primary outcome was acute illness. A month after the final results came in, the primary outcome was changed to laboratory-confirmed COVID-19
19/n Moreover, the outcome that was registered in advance as the primary outcome has completely different results, showing a statistically significant effect for ivermectin but nothing else
20/n Now, there's nothing inherently wrong about switching outcomes, and the authors do mention a reason for changing it, but the fact that it was only changed after the study was finished is very strange
21/n The explanation in the text also doesn't quite make sense. The paper reports excluding people who had a positive serological test at baseline - how can these people have been tested if there were no serological tests when the study started?
22/n So what does this all mean?
Well, overall, it's quite hard to trust the trial's results
23/n The study does not appear to follow the guidelines for implementation and analysis for cluster RCTs, which means that it's hard to know what to make of the analysis ncbi.nlm.nih.gov/pmc/articles/P…
24/n The primary outcome was also switched, with a bunch of other odd inconsistencies in the research that make it a bit hard to know if the conclusions hold water
25/n To their credit, the authors talk about some of these things in the limitations section of the study, but not all of them and I'm not sure they really explain why these are not issues
26/n Anyway, I'm not sure I would rely on this study as evidence for much, despite the large size
27/n Apologies, one of the above tweets is wrong. The authors did indeed take into account the clustering in their statistical analysis
28/n The more I read this paper, the weirder it sounds. So they randomized ineligible people (how?) from two floors that were not clustered, and then assigned them to vitamin C if the other medications were contraindicated?
The weirdest thing about the whole herd immunity through natural infection argument is that it's never happened ever for any disease long-term so it was always a wild idea for COVID-19
Like, sure, pandemics died out - eventually most diseases became endemic and killed only a small number of people each year
But that's definitely what's been bandied about as herd immunity
Imagine if instead of "herd immunity" the message had been "recurring outbreaks with a slowly diminishing fatality rate until after months/years the number of yearly deaths would get low enough to not bother any more"
The study itself is interesting - sleep duration and risk of dementia, lots of follow-up, decent sized sample (although relatively few events) nature.com/articles/s4146…
But the headline is super misleading for so many reasons. My faves:
1. absolute risk is really small (~1 case per 1,000 person-years) 2. The authors acknowledge later in the article that they don't know if this is causal or not
2021 will hopefully be the year that the armchair epidemiologists stop being wrong about infectious disease, excess mortality, etc, and move on to being wrong about something else
Maybe economics?
To clarify, because of course I need to (sigh) this is a joke about the twitter randoms who have deemed themselves experts not a critique of interdisciplinary work
I'm currently working on a paper with 3 economists, an immunologist, a demographer, and 2 statisticians on COVID-19. Non-epis have great and valuable insight!
One interesting point is that this article gets several facts wrong. Whether that detracts from the commentary on science or not is I suppose up to the reader
This statement, for example, isn't really true. The U.S. has had school closures much less severe than (for example) South Korea, or a dozen other places. The reference only talks about Europe!
"Studies have repeatedly concluded" - links to a tweet, and two articles on teacher's unions. There are many studies that have concluded precisely the opposite
"Chronic disease has caused COVID-19 deaths, if we didn't have so much diabetes fewer people would've died" - incredibly dumb argument for many reasons, not least that it is true of LITERALLY ALL HUMAN DEATHS
Yes, if we had solved the biggest medical issue of the modern age fewer people would've died of COVID-19
What of it?
I mean, seriously, we've been trying to 'fix' NCDs for decades, and while they are in theory somewhat preventable they are still a large and growing problem in most places in the world
The basic idea here is that we could be either undercounting or overcounting COVID-19 deaths
I think the most likely explanation is some combination of the two
Based on some very careful examinations of death reporting systems, we can say that there are probably some portion of deaths that are recorded as due to COVID-19 but were not caused by the virus