So, @Covid19Critical does not agree with me, and says that the conclusions of meta-analyses do not change at all once excluding the retracted ivermectin study
Let's go over exactly why I said that removing the study makes a huge difference 1/n
Let's go over exactly why I said that removing the study makes a huge difference 1/n
https://twitter.com/Covid19Critical/status/1416154414654578691
2/n I went over this in my article, which you can find here, but I'll repeat the precise analysis I've done on twitter, because I think it's a fair question
To be clear - I'm not perfect, and not everything I say is right!
https://twitter.com/GidMK/status/1415764372362649601?s=20
To be clear - I'm not perfect, and not everything I say is right!
3/n Judge for yourself whether it is fair to argue that removing Elgazzar largely eliminates the benefit for mortality here, the headline finding and the primary analysis of the paper
4/n So, here's the primary results from the Bryant et al meta-analysis of ivermectin that concluded "Moderate-certainty evidence finds that large reductions in COVID-19 deaths are possible
using ivermectin"
using ivermectin"
5/n Step 1: replicate the findings
While there are very minor differences due to the statistical software used, this is the same Dersimonian-Laird IV model run using the metan command in Stata 15
While there are very minor differences due to the statistical software used, this is the same Dersimonian-Laird IV model run using the metan command in Stata 15
6/n The benefit from this model is huge, very statistically significant, and you can see why the authors made such strong recommendations. There is a significant benefit for mild disease, verging on significance for severe, and overall 60% (!) reduction in death
7/n Step 2: remove only Elgazzar
Immediately, the difference is notable. The primary benefit goes from 60% to 44%, is only marginally statistically significant, and there's no benefit in either Mild or Severe subgroups
Immediately, the difference is notable. The primary benefit goes from 60% to 44%, is only marginally statistically significant, and there's no benefit in either Mild or Severe subgroups
8/n Ok, so that's an issue. What were extremely strong results are now statistically very weak and not beneficial in the major subgroups
But there's a bigger issue here
But there's a bigger issue here
9/n Step 3: add in new research
This study was published shortly after Bryant et al came out. It would certainly have been included in the analysis if it was already out (no fault to the authors of course) so let's add it in
This study was published shortly after Bryant et al came out. It would certainly have been included in the analysis if it was already out (no fault to the authors of course) so let's add it in
10/n The significance of the model entirely disappears. Now ivermectin shows no benefit for mortality at all, even in the overall analysis
11/n But wait, there's more! If you go back to that graph excluding Elgazzar, there's something odd. You see only one study finding a significant benefit for ivermectin
12/n In fact, it seems like the entire mortality benefit for ivermectin, once Elgazzar is excluded, comes down to this one study: Niaee et al
13/n If you look at the Niaee et al preprint, there are quite a number of issues. In fact, @K_Sheldrick noted that the results are worrying, and that the study "should not be included in meta-analyses" 👀👀👀
14/n So, Step 4: Sensitivity Analysis
Rerunning the original model excluding both Elgazzar and Niaee gives you this result
Absolutely no benefit for ivermectin on mortality. CIs include both large benefit and large detriment
Rerunning the original model excluding both Elgazzar and Niaee gives you this result
Absolutely no benefit for ivermectin on mortality. CIs include both large benefit and large detriment
15/n Worth noting that removing these two studies from the analysis takes the I squared statistic down to 0% - in other words, the remaining studies are all statistically consistent with each other
16/n What does this all mean?
In short - once Elgazzar is excluded, the results are incredibly uncertain, and any potential benefit rests entirely on the Niaee study, which at least one expert has argued should never be included in meta-analyses
In short - once Elgazzar is excluded, the results are incredibly uncertain, and any potential benefit rests entirely on the Niaee study, which at least one expert has argued should never be included in meta-analyses
17/n Moreover, because the result is so marginally significant, including a ~single~ new piece of evidence also results in the benefit disappearing even if you still include Niaee in the model
18/n Therefore, in my opinion, this means that excluding Elgazzar removes any certainty and a lot of significance from this analysis
19/n In other words the conclusion - that moderate-certainty evidence found large reductions in death using ivermectin - is entirely reversed. The certainty is gone, and the reduction in death is likely to be very substantially smaller
20/n Indeed, since the benefit now appears to rest entirely on one very worrisome trial, it is hard to see how we can justify any argument other than that we do not have sufficient information to make a conclusion about ivermectin at this point
21/n I remain optimistic that ivermectin will indeed prove to be a "wonder drug" as Elgazzar claimed, but I simply don't think the evidence to date supports that assertion 🤷♂️
22/n With large randomized trials ongoing, we can only wait for them to finish before making a strong judgement as to whether ivermectin is beneficial
It may yet turn out to be fantastic. We simply do not know
It may yet turn out to be fantastic. We simply do not know
• • •
Missing some Tweet in this thread? You can try to
force a refresh
