Thread by @kdpsinghlab

PREPRINT: "Validating a Widely Implemented Deterioration Index Model Among Hospitalized COVID-19 Patients"

Manuscript: medrxiv.org/content/10.110…

Code: github.com/ml4lhs/edi_val…

Why did we do this? What does it mean? Is the Epic deterioration index useful in COVID-19? (Thread)

Shortly after the first @umichmedicine COVID-19 patient was admitted in March, we saw rapid growth in the # of admitted patients with COVID-19. A COVID-19-specific unit was opened (uofmhealth.org/news/archive/2…) and projections of hospital capacity looked dire:

https://twitter.com/umichmedicine/status/1245050056182505473?s=21

https://twitter.com/umichmedicine/status/1245050056182505473

.@umichmedicine was considering opening a field hospital (michigandaily.com/section/news-b…) and a very real question arose of how we would make decisions about which COVID-19 patients would be appropriate for transfer to a field hospital. Ideally, such patients would not require ICU care.

Our hospital's COVID-19 experience and response has been catalogued in great detail in this @wired piece. The state of Michigan was one of the hardest hit in the U.S., and in mid-March, we were planning for the worst with the best possible information.

wired.com/story/in-one-h…

Like many other hospitals, @umichmedicine expanded its cadre of internal medicine physicians in the hospital. Given the rapid deterioration these pts experience, hand-offs b/w clinicians and prioritizing rounds became extra important to identify deteriorating patients.

Our hospital @umichmedicine has many risk prediction tools implemented to support clinical care. Recognizing that we did not have a sufficient sample size to develop a COVID-19 model, we sought to evaluate whether existing models we have implemented could inform decision-making.

One of models we have implemented is the Epic Deterioration Index (EDI), which is a 0-100 score (higher is sicker). The EDI is calculated every 15 mins on hospitalized pts. On our real-time EDI dashboard, we noticed that many of the top EDI scores belonged to pts with COVID-19.

That provided some face validity for the idea that EDI *may* have a role in risk-stratifying COVID-19 pts but the larger question of whether EDI *predicts* clinical deterioration couldn't be answered through simple analyses.

We weren't the only ones. Dr. Ron Li from @StanfordMed also noticed and was the feature of this @statnews piece by @RebeccaDRobbins. Although the EDI is proprietary, it is implemented at >100 U.S. hospitals. Whether it is useful affects many hospitals.

statnews.com/2020/04/01/sta…

Is the EDI valid? Scientifically, the answer is unknown. We were unable to identify any peer-reviewed papers on the validity of the EDI. In a reply to a question by @Bob_Wachter, @SaraMurrayMD noted that @UCSFHospitals found the EDI to perform poorly.

https://twitter.com/saramurraymd/status/1254987698525237249?s=21

https://twitter.com/SaraMurrayMD/status/1254987698525237249

This may very well be true but this finding may also be related to the fact that the EDI fluctuates *a lot*. Individual values don't have much meaning (will describe below), which greatly affects how the score should be interpreted.

Dr. Ron Li from @StanfordMed also noted that trends in the EDI appeared to be useful in a talk at the "COVID-19 and AI Virtual Conference." We found the same for some patients but the fluctuating nature of the EDI score means the slope *also* varies a lot.
hai.stanford.edu/events/covid-1…

It turns out that this sudden interest in EDI did not go unnoticed at Epic. On April 22nd, Epic came out with a press release in which a physician noted that the EDI is "helping save lives." The release also noted ongoing validation work.

epic.com/epic/post/arti…

.@caseymross from @statnews interviewed health centers using the EDI during the pandemic and found disparate EDI thresholds being quoted for how they've used it, including some polar opposite uses. Our study aims to try to answer this question.

statnews.com/2020/04/24/cor…

In their initial press release, Epic included a snapshot of how the EDI appears to clinicians at a patient level.

Note 3 things:
- A score of 37 puts the patient in the "Danger zone"
- The score rises neatly from low- to high-risk
- Score is expressed as a percentage

This figure was updated today shortly after our preprint was released. We will come back to this as it has important implications for how the EDI score should be interpreted.

How does EDI *actually* look to clinicians? Here are 4 example pts (blue line = ICU, red line = mech vent). Ignore color-shading for now. Notice how the top-right and bottom-left panels look similar. Also for top-left, notice how EDI peaked, dropped, and THEN pt needed ICU care.

So is the EDI useful? We studied adult patients admitted with COVID-19 to non-ICU level care at @umichmedicine from March 9-April 7. We used the EDI, calculated at 15-minute intervals, to predict a composite adverse outcome of ICU-level care, mech vent, or in-hospital death.

Among 174 COVID-19 patients meeting inclusion criteria, 61 (35%) experienced the composite outcome. Overall, AUC of EDI was 0.76 (95% CI 0.68-0.84). Patients who met or exceeded an EDI of 64.8 made up 17% of the study cohort and had an 80% PPV for outcome during hospitalization.

Once patients *first had* an EDI ≥ 64.8, median lead time was 28 hours from when this threshold was first exceeded to the outcome. Notice that even if the EDI came down, what matters is that this threshold was *ever* exceeded, even if only exceeded once.

Was the slope helpful? Not as much as the raw score. EDI slope, calculated on a rolling basis, had AUCs of 0.68 (95% CI 0.60-0.77) and 0.67 (95% CI 0.59-0.75) for slopes calculated over 4 and 8 hours, respectively.

In a subset of 109 patients hospitalized for >48 hrs, we also evaluated the ability of the EDI in the first 48 hrs to identify patients at *low risk* of experiencing this composite outcome during their remaining hospitalization.

In this subgroup, the EDI had an AUC of 0.65 (95% CI 0.52-0.77). This drop in AUC is not surprising b/c very high- and low-risk pts were removed. The 13% of patients who *never* exceeded an EDI of 37.9 had a 93% NPV for the outcome for the rest of their hospitalization.

So what are our takeaways? Overall, we found the EDI identifies small subsets of high- and low-risk patients with fair discrimination. The AUCs we found are lower than many described in the literature (e.g. AUC 0.95 w/ 133 pts + 75 predictors) but ours are likely more realistic.

One issue with many existing COVID-19-specific predictive models is that they are developed *and* validated in small populations. This has been catalogued extensively in this @bmj_latest review by @laure_wynants @MaartenvSmeden and others.
bmj.com/content/369/bm…

Another takeaway is that even *small changes* in predictors lead to *large differences* in the EDI because prior normal values are ignored when a new value is obtained. Thus, we recommend the interpretation be based on whether a patient *ever* exceeds specific thresholds.

The substantial variation in EDI also means that recalculation every 15 mins may have diminishing returns. We didn't specifically look at the effect of downsampling the predictions from 15 mins to hourly etc, but I suspect it wouldn't make as big a difference as one might expect.

Let's go back to our earlier notes from Epic's press release. Per the initial release:
- A score of 37 puts the patient in the "Danger zone"
- The score rises nice and neatly from low-risk to high-risk in the spark line
- Score is expressed as a percentage

We found that a (persistent) score of 37 is actually *low* risk (and not high-risk as depicted above). We found that the score *does not* rise and fall neatly. In fact, your peak score is likely the best indicator of your overall risk (and not necessarily your most recent score).

We also found that EDI *should not* be interpreted as a probability. It overpredicts the outcome. Even if you could recalibrate it, you would need a sufficiently large cohort to validate the recalibrated values.

Interestingly, after our preprint was posted this morning, Epic updated their press release to address all 3 of these issues. Here is the new image from their press release.

So were we"using AI before knowing it works" as per the @statnews headline?

- We did inform our hospitalists how to add EDI to their signout and how to interpret the score via @vineet_chopra
- We did suggest using it as an adjunct to help prioritize signout and rounding

But we also had a large team with multiple analysts replicating one another's analyses. For the record, I was on team #rstats.

Others shaped this work substantially: @bnallamo, @jzayanian, Dr. Jenna Wiens, @tsvalley @msjoding and others.

This work also would not have been possible without close coordination between our hospital's COVID-19 operational analytics team, co-led by Carleen Penoza and @VikasParekhMD, members of our Clinical Intelligence Committee, and @UM_IHPI.

Our code and output is available here: github.com/ml4lhs/edi_val…

H/t @darrendahly, whose code I revised to generate calibration plots.

Let me just re-emphasize: This is a preprint. It's not peer-reviewed. Appreciate hearing your feedback and your own experiences with EDI.

Also, @shengpu_tang, one of the PhD students on Dr. Wiens's research team, did phenomenal work on this. He was on team #python. I just realized he is on twitter and wanted to highlight his contributions. He discovered an issue early on with important implications!

One important question brought up by @Bob_Wachter and @SaraMurrayMD is whether whether the EDI score is discriminatory against African Americans in how it is utilized. First, there is the digital divide issue (which hospitals have access to EDI?).

https://twitter.com/bob_wachter/status/1254973204067565568?s=21

https://twitter.com/bob_wachter/status/1254973204067565568

Second, there is the question of whether the raw score itself may not perform well in African Americans. We did not have the sample size to adequately address this question but it's noteworthy that 44% of our COVID-19 pts were African American and EDI did decently overall.

Oops @statsepi tagged the wrong account. Meant to tag you.

.@ronlivs meant to tag you! Didn't realize you were on Twitter.