Thread by @jonc101x

#ChatGPT #AI performance on *open-ended, free-response,* clinical reasoning exams intended for human medical trainees. #LLMs are passing medical licensing exams, but artificial multiple choice designs do not reflect realistic clinical reasoning.
medrxiv.org/content/10.110…

From describing diagnostic schema, generating differential diagnoses and problem lists, to suggesting and interpreting tests, ChatGPT is already demonstrating the surprising ability to often reach a 70% passing threshold on multiple cases.

For simple recall style questions ("describe the typical symptoms of a patient with heart failure,") it will knock those questions out of the park, while showing wide variability and struggling with deeper analytical questions.

This is all for the originally released ChatGPT(3.5). We've had this in the can for two months, under review at a major medical journal, getting last line level edits by the editor. Then... GPT4 (ChatGPT+) released and we have to redo the whole study.🤦Stay tuned.

@DrEricStrong, Alicia DiGiammarino, Isabel Weng, Preetha Basaviah, @PoonamHosamani, Andre Kumar, Andrew Nevins, John Kugler, Jason Hom @StanfordMed @StanfordBMIR @StanfordHospMed @StanfordDeptMed