,
17 tweets,
5 min read
Read on Twitter
1/ #EpiTwitter tweetstorm coming
Warning: strong opinions
The analysis is naive and the findings are ridiculous; the fact that it was published is a sign that when medical journal editors hear "deep learning AI" their brains stop working.
Warning: strong opinions
The analysis is naive and the findings are ridiculous; the fact that it was published is a sign that when medical journal editors hear "deep learning AI" their brains stop working.
2/ first of all, the sheer improbability of the claim.
That you can predict skin cancer in general pop in the next year w AUC 0.9 (and 0.8+ based on medications alone)
That is nuts.
PHOTOGRAPHS OF SUSPICIOUS SKIN LESIONS have worst test characteristics.
That you can predict skin cancer in general pop in the next year w AUC 0.9 (and 0.8+ based on medications alone)
That is nuts.
PHOTOGRAPHS OF SUSPICIOUS SKIN LESIONS have worst test characteristics.
3/ it's as if all previous knowledge of non-melanoma skin cancer epidemiology was wiped out.
Age doesn't matter much. Medications not known to be associated w skin cancer matter a lot.
Many of the meds are taken for years/decades, so couldn't have a huge impact on 1-year risk
Age doesn't matter much. Medications not known to be associated w skin cancer matter a lot.
Many of the meds are taken for years/decades, so couldn't have a huge impact on 1-year risk
4/ Any cancer epidemiologist first thought would be about diagnostic /lead time bias
Non-melanoma skin cancers are asymptomatic, slow growing, underdiagnosed. Any 'prediction" model is more likely to be predicting detection that disease
No discussion. Even in limitations.
Non-melanoma skin cancers are asymptomatic, slow growing, underdiagnosed. Any 'prediction" model is more likely to be predicting detection that disease
No discussion. Even in limitations.
5/ it gets worse. Case-Control with "random" unmatched controls?????
They supposedly have millions of records, find 1,829 cancer patients, then choose 7,665 random patients without cancer to compare to.
The controls are 20 years younger with half the number of conditions! #WTH
They supposedly have millions of records, find 1,829 cancer patients, then choose 7,665 random patients without cancer to compare to.
The controls are 20 years younger with half the number of conditions! #WTH
6/ Then they calculate sensitivity and specificity on this fake population of cases + controls.
YOU CAN'T DO THAT
Sens and spec depend on population prevalence.
You can't create a fake population with 20% prevalence of undiagnosed cancer that will be detected next year!
YOU CAN'T DO THAT
Sens and spec depend on population prevalence.
You can't create a fake population with 20% prevalence of undiagnosed cancer that will be detected next year!
7/ they say incidence of (diagnosis of) skin cancer is about 5 per 100,000 for Asian population.
They are reporting sensitivity and specificity for a "population" that has undiagnosed cancer rate of 20,000 per 100,000.
Why would JAMA Derm accept that?
They are reporting sensitivity and specificity for a "population" that has undiagnosed cancer rate of 20,000 per 100,000.
Why would JAMA Derm accept that?
8/ and why wouldn't you ask for a simple validation test?
Set aside 20% of the whole sample, run "prediction" on entire sample, and report how many who met cutoff actually had cancer diagnosed.
What is going on here?
I think the reviewers & editors were intimidated by AI/ML
Set aside 20% of the whole sample, run "prediction" on entire sample, and report how many who met cutoff actually had cancer diagnosed.
What is going on here?
I think the reviewers & editors were intimidated by AI/ML
9/ the authors spend no time discussing control selection, bias, confounding (THE CONTROLS WERE 20 YEARS YOUNGER, older people take more statins and beta blockers), and LOTS of time talking about sequential multidimensional EMR matrix, convolutional neural network, hidden layers
10/ I can only imagine that reviewers and editors who would have summarily rejected a Case-Control study that found that statins are highly predictive of cancer next year were embarrassed to admit that they didn't understand "TensorFlow backend" and willing to suspend disbelief
11/ and I bet that it's not just journal editors who are failing as gatekeepers
How many VCs are funding healthcare inexperienced entrepreneurs who are waving "deep learning" "ML/AI" claims (that they themselves completely believe) in willful ignorance of what's gone before?
How many VCs are funding healthcare inexperienced entrepreneurs who are waving "deep learning" "ML/AI" claims (that they themselves completely believe) in willful ignorance of what's gone before?
Also this, by @cecilejanssens
Sorry team- I got excited tweeting quickly- totally embarrassing mistake- It's Positive Predictive Value and Negative Predictive Value that depend on prevalence.
That's what has implications for using a test of given sens/spec in different populations.
That's what has implications for using a test of given sens/spec in different populations.
12/ I misspoke about sens/spec earlier (d'oh!). It's Positive Predictive Value that's affected by population prevalence.
example: With Sens 83% and Spec 82%, if prevalence is 0.1% then that means your positive predictive value is 0.4% (8/1808)
example: With Sens 83% and Spec 82%, if prevalence is 0.1% then that means your positive predictive value is 0.4% (8/1808)
13/ Instead, they report a PPV (aka "precision") of 57%
Yes, you can't do that
Yes, you can't do that
14/14 None of this is to say that ML/AI aren't going to be important tools for medicine and public health
At @AledadeACO we're using these tools to predict mortality, no-shows, ED visits, renal failure. But you don't just "plug data into XGBoost"
Epi matters (and we're hiring)
At @AledadeACO we're using these tools to predict mortality, no-shows, ED visits, renal failure. But you don't just "plug data into XGBoost"
Epi matters (and we're hiring)
@AledadeACO If you've made it this far, do you know anyone who'd like to work with me as a senior data analyst? (you?)
Looking for someone who loves epidemiology/methods and is all-in on the @AledadeACO mission (good for patients, good for doctors, good for society)
hire.withgoogle.com/public/jobs/al…
Looking for someone who loves epidemiology/methods and is all-in on the @AledadeACO mission (good for patients, good for doctors, good for society)
hire.withgoogle.com/public/jobs/al…
