Machine Learning and the Cancer-Diagnosis Problem — No Gold Standard: nejm.org/doi/full/10.10…
My latest work in @NEJM with Dr. Gil Welch in which we discuss how machine learning cannot overcome a central problem in cancer diagnosis and we suggest a way forward. #tweetorial 👇🏾👇🏾👇🏾
My latest work in @NEJM with Dr. Gil Welch in which we discuss how machine learning cannot overcome a central problem in cancer diagnosis and we suggest a way forward. #tweetorial 👇🏾👇🏾👇🏾
Machine learning (ML) is a potential powerful tool that may help clinicians deliver faster and more consistent diagnoses and improve patient care. But there are inherent limitations.
So far, most applications of ML in medicine are for visual recognition (e.g. retinal scans, chest x-rays) developed using “supervised learning,” meaning you present the computer with the ground truth and after thousands/millions of data points it gets good at recognizing “truth”
What happens if the “truth” is uncertain? This is a problem in cancer diagnosis which is best illustrated by the inter and intra-pathologist disagreement about what constitutes cancer on a histopathology slides.
Disagreement about the underlying histopathological diagnosis has been documented for prostate, thyroid, breast lesions, and melanoma.
Importantly, given that these cancers have significant problems with overdiagnosis, you could imagine how ML could amplify the problem even further if algorithms are taught based on current standards of what histologically constitutes “cancer.”
Whether it’s the training set, testing set, or tuning, ML algorithms developed will rely on this faulty external standard (human histopathological interpretation). 
To dampened this problem we propose 3 categories (1)“cancer” (all pathologists agree) (2)“not cancer” (all pathologists agree) and (3) an “intermediate” (disagreement among pathologists) category.
This categorization would be efficient, honest, and judicious. This approach should foster
further inquiry into the natural history of intermediate lesions and enable more research on expectant management.
further inquiry into the natural history of intermediate lesions and enable more research on expectant management.
This may not apply in more complex scenarios, or perhaps require more categories (e.g. Gleason score in prostate cancer)
Gil and I hope folks will consider our suggestion when developing ML for histopathologic diagnosis of cancer.
Also, I still can’t believe thoughts out of my brain made it onto the pages of @NEJM, nuts!
And one more thing, I would be remiss not to thank @IAmSamFin and @BenMazer. My engagement with them on and off Twitter helped sharpen my thinking in this space. 🙏🏾🙏🏾🙏🏾
