Day 2. Starting on pitfalls in leveraging EHR by Stephanie HYLAND @ESICM #criticalcare #ehr #datascience #ai #LIVES2022 
This problem is mainly for ML engineers who may not have talked to domain expert or clinicians / end users.
Pitfall 1 : sampling bias
"whos included in the analysis"
"who in your EHR"?
- e.g., - COVID prediction dataset where missing all blood tests were removed, but this missingess has a meaning. Thus not generalisable.
e.g., yesterday I mentioned about females < 6% of sample popn
"whos included in the analysis"
"who in your EHR"?
- e.g., - COVID prediction dataset where missing all blood tests were removed, but this missingess has a meaning. Thus not generalisable.
e.g., yesterday I mentioned about females < 6% of sample popn
Other pitfall: Inability to "time-travel"
e.g., using length of stay outcomes for example.
Pitfall 4: Hazards of measurement frequency
- icu data collected continuously. this is not random!!
- sicker patients got CVP and more frequent ABG
-measurement frequency is informative!
e.g., using length of stay outcomes for example.
Pitfall 4: Hazards of measurement frequency
- icu data collected continuously. this is not random!!
- sicker patients got CVP and more frequent ABG
-measurement frequency is informative!
This means if you build a model based on frequency of lactate modelling, are you telling the clinicians anything 'new' - they know its sick hence doing frequent ABG.
Pitfall 5: overinterpretation of findings
-causal inference from observational data is DOABLE
- but WITH CARE!
Pitfall 5: overinterpretation of findings
-causal inference from observational data is DOABLE
- but WITH CARE!
e.g., if predicted risk according to predictive model increases with blood pressure, it does NOT necessarily imply that reducing blood pressure reduces risk.
(this is what we discussed yesterday about mechanistic effects)
@_hylandSL
@ESICM #criticalcare #ai #ML
(this is what we discussed yesterday about mechanistic effects)
@_hylandSL
@ESICM #criticalcare #ai #ML
@_hylandSL @ESICM Pitfall 6: implementation challenges
-just because you used EHR to build a model doesnt mean it can be implemented on EHR.
- where does the model live?
- how are you going to host?
- what data model ingest and how?
- need informatics team and infrastructure restricted ennviron.
-just because you used EHR to build a model doesnt mean it can be implemented on EHR.
- where does the model live?
- how are you going to host?
- what data model ingest and how?
- need informatics team and infrastructure restricted ennviron.
@_hylandSL @ESICM Summary:
1. sample selection bias
2. imprecision in variable definitions
3. reliance on data not yet collected
4. informative measurement and missingness
5. overinterpreation of findings
6. implementation and EHR integration.
and many more :P
1. sample selection bias
2. imprecision in variable definitions
3. reliance on data not yet collected
4. informative measurement and missingness
5. overinterpreation of findings
6. implementation and EHR integration.
and many more :P
@_hylandSL @ESICM What can you do?
- be careful
- ask many questions
- follow STROPE/ TRIPOD guidelines
-have right team - interdisciplinary, statisticians, ML experts, epidemiologists, people familiar with EHR databae
-DONT DO EVERYTHING YOURSELF !
what an insightful talk @_hylandSL @ESICM #ai
- be careful
- ask many questions
- follow STROPE/ TRIPOD guidelines
-have right team - interdisciplinary, statisticians, ML experts, epidemiologists, people familiar with EHR databae
-DONT DO EVERYTHING YOURSELF !
what an insightful talk @_hylandSL @ESICM #ai
@_hylandSL @ESICM @threadreaderapp unroll
• • •
Missing some Tweet in this thread? You can try to
force a refresh
