Thread by @GidMK

This new systematic review/meta-analysis of ivermectin for COVID-19 has come out, and everyone's asking me to review it

My take - decent study, but the devil's in the details 1/n

2/n The study is here. Because it's about ivermectin, and people are super weird about that specific drug, everyone's going wild with an Altmetric of 8,641 in the week since publication journals.lww.com/americantherap…

3/n I should say that my position on ivermectin previously has been that there is some interesting evidence but that most of the studies are low quality so it's hard to say much (even the one or two high-quality studies aren't very conclusive)

4/n So what about this systematic review?

Well, it was conducted by a group that has some interesting affiliations. At least one member of the FLCCC, an odd pro-ivermectin/vitamin C activist group

5/n That being said, the methodology is pretty good overall. I've got a few qualms, but they're mostly really technical and wouldn't really impact the results much anyway

6/n However, some of the results I think are a bit odd. For example, the authors present this funnel plot of studies, and then say that there was no evidence of publication bias

Except, there kind of is

7/n I don't agree that preprints would necessarily reduce the risk of publication bias in this context - people may be less likely to preprint null results because no one is going to care very much

8/n For example, there's at least one study registered on clinicaltrials dot gov that I can't find results posted for (and does not appear to have been included in this analysis)

9/n Another issue is that I'm not sure I agree with the risk of bias assessment

Now, this is inherently subjective, so take my opinions with a grain of salt, but some of this looks a bit odd

10/n For example, the Elgazzar study. Low or uncertain risk for every element

To me, this study is at a pretty high risk of bias!

11/n Randomization sequence generation is rated at low risk, but here's the information we have on randomization. It contradicts the protocol, which isn't great (they should've had 6 groups, not 2)

12/n Allocation concealment is rated as low risk, but as far as I can tell from the study there's no information on how the allocation sequence was kept concealed except for those two sentences 🤷‍♂️

13/n Technically there's no attrition bias because the authors report 100% follow-up data, but that in and of itself is weird. Usually you get at least a few dropouts!

14/n I could go on. Suffice to say that my assessment of this study would place it at moderate to high risk of bias, especially considering that it's really quite short and there's a fair bit of important information missing

15/n Worth noting that I don't disagree with the assessment for every study - I had a detailed read of Mahmud (2020) and it looks like a very well-done piece of research at a low risk of bias

16/n Another weird one is Niaee - on the risk of bias table, it looks pretty decent, but there are some really odd holes in the study

17/n For example, this was a randomized study of ivermectin for COVID-19, but they included people who were clinically positive and PCR-negative

And somehow, the proportion of PCR negatives in the control group was far higher than in the intervention (47% vs 20%) 👀👀

18/n If the study had a MUCH bigger proportion of people who had PCR +ve COVID-19 receiving the intervention, that's not a great sign wrt the allocation concealment procedures (which aren't that well described either)

19/n This sort of finicky examination is where it gets really, really tough, because like I said, it's a bit subjective. I don't really care either way about the studies included, but people can (and probably will) disagree

20/n That being said, I quickly redid the meta-analysis excluding these two quite weird studies, and the results completely change, with no significance at all (RR 0.72, 0.41-1.25)

Make of that what you will

21/n Even if you ignore my opinions, limiting the analysis to only studies rated BY THE AUTHORS as "low" risk of bias gives you a completely inconclusive result (RR 0.65, 0.12-3.59) 🤷‍♂️

22/n So I don't really have an issue with the methodology or the results of the study, except to say that I think the risk of bias here is generally quite high and publication bias is likely

The conclusions however...

23/n I don't think this is justified at all to be honest. Even if we ignore the issues that I mentioned in this thread, most of these studies are really short (probably because they're all preprints) containing only a fraction of the information you need to properly assess them

24/n There's a good reason the risk of bias for so many regions here is 'unclear' - the studies simply don't include enough information! Often the methods sections are 3-4 paragraphs long (v. short for an RCT)

25/n Moreover, publication bias looks to be a bit of an issue, and it's not really addressed. Quite likely to be some null findings that are floating around but will be published eventually!

26/n On top of this, if you only look at studies that the authors have rated as having a low risk of bias for all domains, the evidence is pretty much squarely null, although this is limited by the fact that there are only 3 good studies

27/n I don't think that this study really supports the use of ivermectin as a treatment for COVID-19 outside of a clinical trial, which is a shame because it'd be fantastic to have another effective drug for the disease

28/n The depressing thing is that 100s of millions of people, mostly in developing countries, have had ivermectin for COVID-19, and yet we've only got 3 low-risk studies with <700 participants total to use as good evidence 🤯🤯🤯

29/n Oh, one other point is that the authors considered the Fonseca study an "outlier" because exclusion lowered the I^2, which I don't understand because excluding the Elgazzar study lowers the I^2 a lot as well 🤷‍♂️

30/n Also, I should reiterate that while I disagree with the conclusions, the study itself is pretty strong. The methodology was well constructed, I have few if any criticisms of the actual research the authors conducted