THREAD: Two bits of excellent #COVID19 news today. First, remdesivir significantly shortened disease in a 800-patient randomized placebo-controlled trial run by NIAID. Details will come later. Even more interesting though... (1/n)
statnews.com/2020/04/29/gil…
statnews.com/2020/04/29/gil…
Remdesivir also came very close to showing significantly shortened disease in the randomized control trial from Hubei, China, of 237 severe patients (18 vs 23 days). Detailed data were published today in the Lancet. Why is this more interesting? (2/n)
thelancet.com/journals/lance…
thelancet.com/journals/lance…
The Hubei data are interesting because they show an effect that is (in my deductive scientific opinion) clear but that by the rules does not exist. How can I claim something is scientifically clear when it does not exist? The answer has to do with intuition vs sensing... (3/n)
Clinical improvement rates in the Hubei-Lancet study for all 237 patients are shown in the Kaplan-Meier chart below. Remdesivir vs placebo had a hazard ratio for clinical improvement of 1.27 with 95% confidence interval 0.89 to 1.80. What the heck does this mean? (4/n) 
The curve shows the fraction of patients improved on 2 points in a severity scale or discharged from the hospital over time. The hazard ratio (poor term, really a benefit ratio) of 1.27 means you are 1.27x more likely to improve on remdesivir. So what's not to like? (5/n)
What's not to like is the 95% confidence interval, the range of hazard ratios accounting for 95% of statistically possible outcomes, is 0.89 to 1.80. This number overlaps 1, which would be no benefit. This makes the trial officially a failure. But is it really? (6/n)
A strict statistician would say we can't make any conclusions from the trial, it was a failure and the 1.27x number means nothing. I say not so fast; the trends can mean something. First, we must remind ourselves why we require effects to be 95% statistically likely... (7/n)
The 95% convention means we accept that 5% of conclusions from statistics studies will actually be wrong. The field has decided not to go lower to say 90% because having 10% of conclusions needing future retraction seems too high. However we should remember that... (8/n)
A benefit ratio of 1.27 with a 95% confidence interval of 0.89 to 1.80 still means more likely than not there is a real benefit, and the best estimate of that benefit is 1.27x. In addition, the benefit ratio improves for patients who start treatment earlier (graph)... (9/n)
This is as expected for an antiviral medication, so by fitting expectations is somewhat reassuring. I am not saying we can call this trial a success, but it's also scientifically and logically invalid to call it a complete failure. Why do I bother going through this? (10/n)
The Hubei-Lancet study would be a moot point if the NIAID study does indeed meet its endpoint with >95% statistical confidence. I bring this up because clinical trial near-misses are almost always described as complete failures, which is confusing and can be harmful... (11/n)
The NY Times for example said today that the Hubei-Lancet trial "reported that the drug offered no benefit to severely ill patients in China". NOOO!!! The trial did not report that. It reported that the benefit was not statistically significant... (12/n)
nytimes.com/2020/04/29/hea…
nytimes.com/2020/04/29/hea…
This is a common error of description by medical journalists (and also by some doctors) but we MUST stop making it. It creates the misimpression that there are contradictory trial results, which causes people confusion and distrust of actual positive results... (13/n)
Conflating near-misses of statistical significance with no effect is failing to realize that statistics is about degrees of certainty, not about black and white. This confusion can also have the effect of creating false equivalences between near misses and direct hits... (14/n)
Because statistics is about interpreting uncertainty, it helps to bring other scientific approaches such as an understanding of hypothesis and mechanism to interpretation of statistical results. Yes it's messy but the world is messy... (15/n)
BTW when WHO leaked the Hubei trial top-line results, Gilead said "'trends in the data suggest a potential benefit for remdesivir, particularly among patients treated early in disease" which appears correct (16/n) statnews.com/2020/04/23/dat…
Finally it is worth noting the trial meant to enroll 400 patients but only got 237. With 400, the benefit may have been statistically significant (confidence intervals shrink with more samples). Thus one could say the trial failed the drug, not the drug failed the trial. (n/n)
