,
16 tweets,
8 min read
Read on Twitter
I had the pleasure of speaking as part of an AI tutorial at ARVO today. Lots of really great work going on in the field, causing me once again to agree with @EricTopol's take on this.
A few people asked for my slides, so figured I'd make a thread out of it. 1/
A few people asked for my slides, so figured I'd make a thread out of it. 1/
Full (22) slides are here sgfin.github.io/assets/slides/…
The primary audience was practicing ophthalmologists who are interested in AI -- hence all the eye imaging cases. But I hope it's interesting more broadly.
Highlights below. 6 questions we must learn to ask:
2/
The primary audience was practicing ophthalmologists who are interested in AI -- hence all the eye imaging cases. But I hope it's interesting more broadly.
Highlights below. 6 questions we must learn to ask:
2/
Question #1: How might our model be tainted with information from the future? 
Label Leakage!
Question #2: How might our test set be contaminated with information from our training set?
Train-test set leakage!
Question #3: How might our model be confounded?
Healthcare dynamics as well as patient confounders!
See arxiv.org/abs/1811.03695 and bmj.com/content/361/bm…
See arxiv.org/abs/1811.03695 and bmj.com/content/361/bm…
Question #4: Is our model performance consistent across patient subpopulations?
(Q 4a: Why isn't our bias going away with more data?)
(Q 4a: Why isn't our bias going away with more data?)
Discrimination is not a single problem.
See @irenetrampoline and @david_sontag's brilliant arxiv.org/abs/1805.12002 and papers.nips.cc/paper/7613-why…
See @irenetrampoline and @david_sontag's brilliant arxiv.org/abs/1805.12002 and papers.nips.cc/paper/7613-why…
Question #5: How might the data we feed our model change over time? (And why do we care?)
Question #6: Can we anticipate any unintended consequences?
(Too many to list given limited time: Overdiagnosis, Unequal Distribution, Adversarial attacks, job disruption, etc.)
(Too many to list given limited time: Overdiagnosis, Unequal Distribution, Adversarial attacks, job disruption, etc.)
Conclusion:
DL may mean we don't need experts for *feature engineering* in the way we used to, but *clinical judgment* (providers) and *clinical research* judgment (biostats, epidemiologists, trialists, etc) is as needed as ever.
Both MLers and MDs need humility, partnership.
DL may mean we don't need experts for *feature engineering* in the way we used to, but *clinical judgment* (providers) and *clinical research* judgment (biostats, epidemiologists, trialists, etc) is as needed as ever.
Both MLers and MDs need humility, partnership.
Addendum:
I was extremely impressed by the team at @EyeDiagnosis, of the first autonomous clinical AI system. These folks are the real deal, and are really leading the charge in an admirable way. (Underappreciated, IMO)
Also great seeing @DeepMind_Health, @GoogleAI, others!
I was extremely impressed by the team at @EyeDiagnosis, of the first autonomous clinical AI system. These folks are the real deal, and are really leading the charge in an admirable way. (Underappreciated, IMO)
Also great seeing @DeepMind_Health, @GoogleAI, others!
@EyeDiagnosis @DeepMind_Health @GoogleAI (Others includes @pearsekeane, @MichaelFChiang, @DeepMindAI.)
Also want to thank Michael Abramoff for the invite and @Emily_Alsentzer and @zakkohane and rest of lab for feedback on slides.
Also want to thank Michael Abramoff for the invite and @Emily_Alsentzer and @zakkohane and rest of lab for feedback on slides.
