Thread by @GidMK

NEJM, arguably the biggest medical journal in the world, has just published this study stating that "remdesivir may have clinical benefit in patients with severe Covid-19"

A quick skim says to me that the paper is...bad

Let's do a live twitter critical appraisal and see! 1/n

https://twitter.com/NEJM/status/1248697013870493698

Ok, first things first - when was the study submitted?

NEJM doesn't publish this in an easily findable format, but the PROTOCOL for the study was written on the 21/03, and based on the author's disclosures it was submitted around the 02/04

Published 10/04. Quick turnaround!

Ok then, who funded this massive, viral (excuse the pun) study?

Gilead! A medical company who makes the drug - remdesivir - that was studied

And going back to those author disclosures, it seems that about half (!) of the 60-odd authors were either directly employed by or owned stock in Gilead during this investigation

That's a pretty big conflict of interest

Good that they disclosed it though! Much better than keeping this all hidden

Ok, so, what did the authors actually do?

Well, they took a bunch of people who were given remdesivir while very sick with COVID-19 and followed them up for at least 1 day after

Given that the protocol is dated a month after the enrolment of the last patient, we can assume that this was a RETROSPECTIVE trial

I.e. all of this treatment was done, then the authors decided to look at the data

Nothing wrong with retrospective observational trials, but it means we have to be REALLY cautious about causal conclusions

The study design was very simple - extremely sick people were given the drug on compassionate grounds (i.e. we don't know if it works but they might die anyway), and then their records were examined to see what happened

Halfway through the methods, we get this wonderful gem

It seems the paper was ghostwritten by a Gilead employee (who is NOT an author)

That's an amazing thing to have published in a @NEJM paper!!!

Don't get me wrong, it happens all the time, but rarely is it acknowledged so blatantly in the text

It also appears to go against @NEJM editorial guidelines which...isn't great?

Statistical analysis appears reasonable, except for this gem

Seems like they're basically saying "because the analysis method we chose couldn't easily accommodate best practice, we didn't do it"

Given that this was a RETROSPECTIVE study, it seems likely that they could've just...used a different analysis methodology if they thought that multiple comparisons would be an issue?

Weird

Anyway, on to the results, and this delightful first paragraph

Titled "patient randomization" but doesn't talk about randomization (because there wasn't any)

The perils of extra-short peer review perhaps?

The paragraph also isn't good news for the study's rigor

Of 61 total patients, 8 were excluded because of missing information

Of the remaining 53, only 40 received the full course of the drug

Worth pointing out at this point that there's NO CONTROL GROUP

That's a huge problem for inference - how do we know if any improvement seen in this trial had anything to do with the drug?

On to the patient demographics

It's a relatively young, although fairly unhealthy population

About what you'd expect given the inclusion criteria (although a bit younger perhaps)

And here we get to the main results

Of the patients (53) treated with remsevidir, most improved!

The death rate also appears to have been pretty low at only 13% (7/53)

Remember, these were VERY sick people. In groups admitted to an ICU for #COVID19, the median death rate is usually around 30%

The authors have constructed an arbitrary ordinal scale here from 1-6, where 6 is the worst (death) and 1 is the best (discharged)

Based on this scale, most patients improved on remdesivir

HOWEVER, there was a huge issue here

Most patients didn't have the follow-up required to perform this calculation. In fact, based on the IQR presented here, less than 25% had 28 days follow-up data to analyze!

Younger people did better than older, people who were not mechanically ventilated did better than those who were (not surprising perhaps)

There were also a large number of reported side-effects, although given the lack of a control group and how sick these people were it's very hard to know if they had anything to do with remdesivir

As a fun statistical point, the confidence intervals for some of these regression analyses were, uh, pretty wide

Older patients had between 35% and 9417% increase risk of death!

So what does this all mean?

Well, the authors talk about it in their discussion

Apparently, the mortality rate was lower than expected, which is "noteworthy"

Now, I'd argue that this is...problematic

It is extremely difficult to compare patients across trials, and absolutely NOT best practice

We also saw a high dropout rate in the trial, with 20% of patients not receiving the complete treatment!

https://twitter.com/GidMK/status/1249547200834580485?s=20

We also have very few patients in this trial, and no control group

Also, the patients were selected by their doctors - perhaps picking the patients that they thought had a fighting chance?

We can't really say whether the death rate was low or high from the data we have!

To the author's credit, the final paragraph acknowledges most of this!

Let's sum up:

❌very small retrospective trial
❌no control group
❌written by pharma funder
❌high dropout
❌short timeframe
❌missing data
❌poorly written/edited
❌somewhat odd stats
❌highly selected patient cohort

✅no causal conclusions!

Basically, it was a very small study with HUGE caveats that showed an interesting possibility

Hard to say anything more than that without a proper trial of some kind

Some might argue that this should not have been published as a research trial, given the many caveats and huge conflicts of interest this study seems to contain

I guess that's a question for @NEJM, who appear to have garnered millions of reads on the article in the last few days