Thread by @AdamRodmanMD

Can GPT-4 solve really hard medical cases and come up with a good list of differential diagnoses?

@zahirkanjee @byrondcrowe and my study is out in @JAMA_current , and the short answer is, “Yes.”

But what does this all mean? 🧵⬇️

https://twitter.com/JAMA_current/status/1669366653300211712

First, the topline results – in running all of the post-2021 published @NEJM clinicopathologic conferences, ChatGPT got the final diagnosis in 39% of cases, and had the final diagnosis in its differential in 64% of cases.

And using a previously-published scale of “quality,” there were good differentials.

Are these numbers good? Most of us don’t want a doctor who’s right only 40% of the time.

I think the answer is yes.

These cases are INCREDIBLY difficult. I’m not aware of any literature looking at the actual rates of humans solving CPCs, but the very brave authors of this study (bmcmedinformdecismak.biomedcentral.com/articles/10.11…) solved them together and only got 28% right.

ChatGPT performs just as well as the top commercial differential diagnosis generators, and that’s with no special medical training (of course we don’t even KNOW GPT-4's training). It also develops more “useful” lists of differentials, accepting the subjectivity of that judgment.

So what does this mean? Will you be going to see Dr. GPT soon? After all, we started looking into this at the same time me and @ZSchoepflinMD hosted Dr. GPT as an expert discussant at the BIDMC_IM CPC

https://twitter.com/AdamRodmanMD/status/1646971534135947264

The answer is “no” (or rather “not yet.”) CPCs are best thought of as very, very hard, and very, very nerdy puzzles. The data is all curated by a human expert, for other humans to solve (and learn from, which is their primary purpose).

Real life diagnosis is not like this at all – data has to be gathered via a hypotheticodeductive process and then mentally sorted and prioritized with considerable amounts of epistemic uncertainty @andrewolsonmd

https://twitter.com/andrewolsonmd/status/1669471789880078336

One thing I think our study DOES suggest is that doctors who are “stumped” on their cases right now could give GPT-4 a problem representation in order to broaden their differential (eg, using it for debiasing). Which does appear to be how a lot of people are using it already.

And of course, I have to point out that ChatGPT is *NOT* HIPAA complaint.

Now for the fun part – the history! Where does our study fit into the larger literature? And what are the next steps?

Doctors have been trying to build machines that think like doctors for over a century. I love to show this quote to people (h/t @Rochalimaea because it’s from 1918!!!

But the ostensible beginning of the mission to build a diagnostic artificial intelligence started in 1959, with the famous paper by Ledley and Lusted in Science – The Reasoning Foundations of Medical Diagnosis (science.org/doi/10.1126/sc…)

It is a difficult read, but it basically set out a roadmap to build a diagnosis generator – using epidemiological data to establish the pre-test probability, and then seeing how different findings, tests, and symptom complexes affect the post-test probability.

And they used the NEJM CPCs as their model of showing this was possible.

Since 1959, there have been dozens of “differential diagnosis generators”. The most famous was INTERNIST-I in the 1970s and 1980s, which tried to model the mind of @PittDeptofMed Jack Myers (nejm.org/doi/full/10.10…)

(also love that the headline is "Artificial Intelligence Comes of Age" from 1978! another @Rochalimaea hobby of finding how repetitive our narratives are about technology and AI)

And differential generators are still around today – the most widely used and studied in Isabel (isabelhealthcare.com). And when these things are studied, they use (drumroll please) NEJM CPCs!

TL;DR #1 – solving complex cases (in particular, NEJM CPCs) has been a standard – a somewhat arbitrary one – for testing differential generators since the concept was developed. In this sense, ChatGPT performs with the best of them.

What about the LLM literature? Google and OpenAI have talked about the success of their LLMs in terms of their ability on multiple choice question tests (like passing the USMLE pubmed.ncbi.nlm.nih.gov/36812645/ or failing GI journals.lww.com/ajg/Abstract/9…)

Is this a good benchmark?

Of course not!

The ability to pass a standardized test tells us very little about how it makes decisions (even though the medical profession still loves them). And I’m not the only one who thinks that -- eg, @peteratmsr

https://twitter.com/peteratmsr/status/1659954084638769152

Should solving CPCs be a good benchmark? It's what we’ve done for over 60 years. And even if it’s better than MCQs, I think it’s time to move on (and yes, I realize the irony, since I did this study!)

Any medical professional who has dabbled with GPT-4 more than a little bit knows that it can show uncanny insights about clinical reasoning – comparing and contrasting illness scripts and seeming to understand Bayesian reasoning and how new information changes probabilities.

Ledley and Lusted had a very mechanistic understanding of how diagnoses were made. But since that time, we’ve learned so much more about clinical reasoning, especially from cognitive psychology.

I personally think that this technology is advanced enough RIGHT NOW that we need to move on from studying it like a differential generator. We need to start studying actual reasoning processes and developing new benchmarks @BageLeMage

This is important because soon – not now, and probably not with GPT-4 – there are going to be studies with LLMs doing decision support with real patient data. And so much more goes into reasoning than just providing a rank-ordered differential.

So TL;DR #2 – our @JAMA_current study is at an intersection of the differential generator literature, and a new medical LLM literature. And it underscores the need to start thinking critically about how we’re going to evaluate these systems.

For those of you still reading, I came at this from an unusual angle -- I'm a doctor and educator, but I'm also a historian with a focus on the development of diagnosis.

I host a history podcast with @ACPinternists called @BedsideRounds (bedsiderounds.org)

I have a whole series on the development of diagnosis (with three episodes covering CDS and the birth of AI in the next two months):

1: bedside-rounds.org/episode-59-cry…
2: bedside-rounds.org/episode-63-sig…
3: bedside-rounds.org/episode-64-a-v…
4: bedside-rounds.org/episode-68-the…
5: bedside-rounds.org/episode-69-the…

6: bedside-rounds.org/episode-72-pro…

And if you're a member of @ACPIMPhysicians you can get CME/MOC credits just for listening at acponline.org/BedsideRounds