,
13 tweets,
4 min read
Read on Twitter
Skin cancer better diagnosed by deep learning than doctors aiin.healthcare/topics/diagnos… Press releases like these really irritate me. 
Let’s a take a dive into this study. Here is a link to it: ejcancer.com/article/S0959-…
First of all, this study was not about skin cancer writ large, but about melanoma, specifically. I think that’s an important distinction.
Researchers trained a convolutional neural network (CNN) using a total of 595 images that were either classified as benign nevi or melanoma.
What they found is that the CNN achieved a mean sensitivity/specificity/accuracy of 76%/60%/68% over 11 test runs. In comparison, the 11 pathologists achieved a mean sensitivity/specificity/accuracy of 51.8%/66.5%/59.2%. See ROC curve:
Does this mean that machines are now better than pathologists? Not so fast…
What is CRITICAL to understand is how the study was constructed. The images used in this study were image sections (0.06% of the whole slide on average) with a 10-fold magnification which were randomly cropped (one crop per slide/patient) Like this:
This does not reflect the everyday workflow of a pathologist who can scan an entire slide, zoom in and out, and may even have some clinical information.
These pathologists were handicapped from the start!!!
Also, I would have liked to have see the distribution of melanoma stages which were evaluated. Histopathologic disagreement between malignant vs benign in the diagnosis of early stage melanoma is extremely high, as seen in this 2017 BMJ paper: bmj.com/content/357/bm…
Both INTRAobserver and INTERobserver concordance among pathologists in the diagnosis of melanoma is POOR.
What pathologists have trouble with the most is agreement in early melanoma, and I think this is the area that AI could shine, MAYBE. There is a lot more work to do.
I think AI has a place in dermatology and pathology, but these "head to head" comparisons do not reflect real world conditions.
