Thread by @GidMK

This is one of the larger trials of hydroxychloroquine, ivermectin, povidone-iodine and zinc for prevention of COVID-19

At first, it appears positive, but there are some worrying issues

Some peer-review on twitter 1/n

2/n The study is here, and it's a cluster-randomized controlled trial, where people living in dorms of Singapore were given one of the 4 treatments or a vitamin C control during large COVID-19 outbreaks in the dorms ijidonline.com/article/S1201-…

3/n The results seem to show that people who take HCQ or P-I have fewer infections than those who only have vitamin C, with a really impressive risk reduction

4/n If these reductions are true, either HCQ or P-I would be a massive, world-changing reduction in risk for the prevention of COVID-19 (although zinc and ivermectin would be largely useless)

5/n But reading the methodology, some things immediately spring out. Firstly, this was a cluster-randomized trial, where participants were randomized to the intervention by the floor of their dorm

6/n ...but this was NOT what the study was originally designed for. The sample sizes, statistical analysis, and other protocols were designed for a parallel-arm clinical trial

This is not ideal

7/ While cluster-randomized trials are not my forte, all of the statisticians I've asked so far have said that turning a 5-arm study into a cluster RCT in this way leaves you with quite a lot of issues

7.5/n In particular, by clustering at the floor level the researchers essentially reduced their effective sample size from 3,000 patients to 40 floors. You have to take this into account in the statistical analysis, which the study does not seem to have done

8/n There's also an issue with how COVID-19 was ascertained in the study. Patients gave a blood sample at day 1 and 42, and then medical records were searched to see if they'd had a positive PCR between those two dates

9/n So, firstly, that is an issue because there's a bias in the PCR testing - only those who had a test (for whatever reason) AND had it entered into whichever medical records system would show up as positive

10/n On top of that, the serology isn't great. Serology often misses people who are tested early in their infection, so the estimates of infection rates for all the groups are probably too low

11/n If you correct for this, instead of the percentages given in the results, you get much higher values for all the treatments, and a potentially reduced protective effect

12/n Also, the study excluded people who already had COVID-19 antibodies at the start, but there isn't much information about these people, which is an issue given that they were excluded after already being randomized (and treated?)

13/n Moreover, a full 20% of the sample was excluded because they were not from randomized floors, which is bizarre. How were they recruited, randomized, and apparently treated if they were not on randomized floors?

14/n It seems very much like this trial was conceived, planned, AND RUN as a randomized parallel-arm trial and then halfway through switched to a cluster RCT

15/n Ok, so that's all pretty worrying. The results may not be statistically viable, the methodology has quite a few flaws

But there's actually a potentially bigger issue

16/n The primary outcome reported in the trial was laboratory-confirmed (through either PCR or serology) COVID-19. This is the measure that the headline results is based on

17/n But if we go to the pre-registration for the study (which, incidentally, doesn't talk about controlling for a clustered design), there's something a bit weird

The primary outcome was originally "acute respiratory illness"

18/n So when the study was registered, in June 2020, the primary outcome was acute illness. A month after the final results came in, the primary outcome was changed to laboratory-confirmed COVID-19

19/n Moreover, the outcome that was registered in advance as the primary outcome has completely different results, showing a statistically significant effect for ivermectin but nothing else

20/n Now, there's nothing inherently wrong about switching outcomes, and the authors do mention a reason for changing it, but the fact that it was only changed after the study was finished is very strange

21/n The explanation in the text also doesn't quite make sense. The paper reports excluding people who had a positive serological test at baseline - how can these people have been tested if there were no serological tests when the study started?

22/n So what does this all mean?

Well, overall, it's quite hard to trust the trial's results

23/n The study does not appear to follow the guidelines for implementation and analysis for cluster RCTs, which means that it's hard to know what to make of the analysis ncbi.nlm.nih.gov/pmc/articles/P…

24/n The primary outcome was also switched, with a bunch of other odd inconsistencies in the research that make it a bit hard to know if the conclusions hold water

25/n To their credit, the authors talk about some of these things in the limitations section of the study, but not all of them and I'm not sure they really explain why these are not issues

26/n Anyway, I'm not sure I would rely on this study as evidence for much, despite the large size

27/n Apologies, one of the above tweets is wrong. The authors did indeed take into account the clustering in their statistical analysis

28/n The more I read this paper, the weirder it sounds. So they randomized ineligible people (how?) from two floors that were not clustered, and then assigned them to vitamin C if the other medications were contraindicated?