My first ever tweetorial! In the last 48 hours, there has been a lot of buzz in the lay press (WSJ, CNN) about a new observational study of hydroxychloroquine (HCQ) in COVID-19 (ijidonline.com/article/S1201-…).
In case you haven't seen it, the single-center study demonstrates a very strong association between early treatment with HCQ for hospitalized patients and a substantial (50% or greater) reduction in the risk of 28-day mortality.
I don’t think I can match @ProfDFrancis in the use of clever GIFs and polls. But my main goal here is to educate people about all the types of issues that come up in observational comparative effectiveness studies.
Of note, there is nothing really profound here—just a bunch of fairly obvious issues. And I’m not even going to mention the completely obvious one-- confounding by indication. (4/x)
Also, I know that the Henry Ford group who published this are well-meaning. In fact, they were one of the first groups to start an RCT testing prophylactic HCQ, which is still ongoing. (5/x)
I am hopeful that they’ll be able to complete enrollment in that trial soon so that we can add to the scientific literature on HCQ in COVID-19. In the meantime, here are my basic concerns about the paper… (6/x)
(1) Immortal time bias—median time to starting HCQ was 1 day (IQR 1-2), and the KM curves show that by day 2 there was already an 8% absolute mortality benefit with HCQ, which increased to about 11% on day 3. (7/x)
(2) Competing risks-- follow-up was in-hospital only but the authors failed to incorporate discharge to home as a competing risk (same issue as in the single-arm compassionate use Remdesivir trial published earlier this year in NEJM). (8/x)
(3) Throwing away lots of perfectly good data. As best I can tell, the authors adjusted for 18 covariates, all of which were coded in a binary fashion including age, BMI, serum creatinine, and admission oxygen saturation. Why would anyone do this? (9/x)
4. Inclusion of post-admission covariates in risk-adjustment and the propensity score including need for ventilator, ICU admission, and use of steroids and tocilizumb at any time during the admission. They could have easily included these as time-varying covariates. (10/x)
5. Exact matching on the propensity score in the propensity-matched analysis, which seems to have resulted in completely identical patient populations with respect to the covariates of interest. In addition to producing a truly remarkable Table 3/Love Plot. (11/x)
Quite honestly, I can’t figure out a good reason why anyone would have done this at all. At a minimum, it probably results in a serious loss of power. (12/x)
6. Failure to account for date of admission or to report use of HCQ over time. Presumably, increased use of HCQ occurred later in the series—a time at which the health system had learned how to care for these patients much better. (13/x)
The authors argue that these results show that the benefit of HCQ is highly dependent on the timing of administration, because previous studies of severely ill patients (RECOVERY) and of very early administration (post-exposure prophylaxis in the Boulware et al NEJM)... (14/x)
... have both shown no benefit. However, if the benefit seen in the Henry Ford study is real, this would represent the narrowest therapeutic range I’ve ever seen. (15/x)
There probably are others that are more subtle as well, but these were all fairly obvious. Quite frankly, any competent peer reviewer should have picked up on several of these, so a lot of my issue here is with the journal (International Journal of Infectious Diseases). (fin)
