1/
Welcome to a new @MayoMN_IMRES #EBM #Tweetorial Series!
Content is brought to you by @CourtHarrisMD @Jaegermeister58 & @ColinWestMDPhD. Our purpose: educate on confusing #EBM topics! Engage with us & RT if you find this helpful!
Topic: Clinical vs statistical significance
Welcome to a new @MayoMN_IMRES #EBM #Tweetorial Series!
Content is brought to you by @CourtHarrisMD @Jaegermeister58 & @ColinWestMDPhD. Our purpose: educate on confusing #EBM topics! Engage with us & RT if you find this helpful!
Topic: Clinical vs statistical significance
2/
Let’s get our definitions straight first!
➡️Clinical significance: the importance of the observed effect in practice (i.e., is the magnitude of the effect large enough to affect clinical care)
Let’s get our definitions straight first!
➡️Clinical significance: the importance of the observed effect in practice (i.e., is the magnitude of the effect large enough to affect clinical care)
3/
➡️Statistical significance: p < 0.05 (commonly). This does NOT mean the probability the conclusion is false is <5%!
See this article from @Amstatnews on guiding principles for p-value use (this is a game changer article people): amstat.tandfonline.com/doi/full/10.10…
➡️Statistical significance: p < 0.05 (commonly). This does NOT mean the probability the conclusion is false is <5%!
See this article from @Amstatnews on guiding principles for p-value use (this is a game changer article people): amstat.tandfonline.com/doi/full/10.10…
4/
To quote the article:
Q: Why do so many colleges and grad schools teach p = 0.05?
A: Because that's still what the scientific community and journal editors use.
Q: Why do so many people still use p = 0.05?
A: Because that's what they were taught in college or grad school
To quote the article:
Q: Why do so many colleges and grad schools teach p = 0.05?
A: Because that's still what the scientific community and journal editors use.
Q: Why do so many people still use p = 0.05?
A: Because that's what they were taught in college or grad school
5/
Do NOT stop thinking when you see a p<0.05! Many questions should arise before you accept conclusions including:
✅Are the results valid? Appraisal of validity and risk of bias is a core #EBM skill, which we will review separately in the future.
Do NOT stop thinking when you see a p<0.05! Many questions should arise before you accept conclusions including:
✅Are the results valid? Appraisal of validity and risk of bias is a core #EBM skill, which we will review separately in the future.
6/
Let’s look at an example to illustrate:
The effect of statins on carotid intimal thickness:
pubmed.ncbi.nlm.nih.gov/15530920/
Let’s look at an example to illustrate:
The effect of statins on carotid intimal thickness:
pubmed.ncbi.nlm.nih.gov/15530920/
7/
This study was a meta-analysis with results showing a beneficial effect of statins (p<0.00001), however the weighted mean difference was -0.02235 [-0.02656, -0.01614] (measured in mm/y).
Wow folks, look at that tiny p-value!
This study was a meta-analysis with results showing a beneficial effect of statins (p<0.00001), however the weighted mean difference was -0.02235 [-0.02656, -0.01614] (measured in mm/y).
Wow folks, look at that tiny p-value!
8/
BUT …is 0.02 mm/year clinically important?
Does this connect with cardiovascular outcomes, or mortality, or other things we and our patients care about directly?
BUT …is 0.02 mm/year clinically important?
Does this connect with cardiovascular outcomes, or mortality, or other things we and our patients care about directly?
9/
Now that we have you thinking, here are further factors to consider!
✳️Check out Table 1! It is important to see if study subjects are representative of the population and/or similar to your patient.
Now that we have you thinking, here are further factors to consider!
✳️Check out Table 1! It is important to see if study subjects are representative of the population and/or similar to your patient.
10/
✳️What is the magnitude of the effect? Do they have the NNT listed/can you calculate it? An acceptable NNT depends on costs and adverse effects.
✳️What is the magnitude of the effect? Do they have the NNT listed/can you calculate it? An acceptable NNT depends on costs and adverse effects.
11/
✳️Did the study measure and report and report an endpoint that is valued by your patient? Don’t assume all patients prefer the same outcome.
✳️Did the study measure and report and report an endpoint that is valued by your patient? Don’t assume all patients prefer the same outcome.
12/
✳️Was the endpoint composite?
If yes, must see which component primarily affected results. Typically these will be softer, less impactful.
✳️Was the endpoint composite?
If yes, must see which component primarily affected results. Typically these will be softer, less impactful.
13/
✳️Was the outcome a surrogate?
If yes, is this a valid predictor of your true outcome? Could there be other known or unknown contributors at play in this disease process?
✳️Was the outcome a surrogate?
If yes, is this a valid predictor of your true outcome? Could there be other known or unknown contributors at play in this disease process?
14/
✳️Watch out for trials with large treatment effects, but few numbers of events. This is true especially in trials that are stopped early.
Check out this excellent paper by @vmontori: pubmed.ncbi.nlm.nih.gov/16264162/
✳️Watch out for trials with large treatment effects, but few numbers of events. This is true especially in trials that are stopped early.
Check out this excellent paper by @vmontori: pubmed.ncbi.nlm.nih.gov/16264162/
15/
✳️Beware when an effective comparator arm isn’t used, and the control is no treatment/placebo.
✳️Beware when an effective comparator arm isn’t used, and the control is no treatment/placebo.
16/
Our takeaway point & #EBMpearl: yourself:
➡️Is this result credible?
➡️Will this result impact my patients?
Our takeaway point & #EBMpearl: yourself:
➡️Is this result credible?
➡️Will this result impact my patients?
17/
Thank you for joining us for our first brief #EBM #Tweetorial! Look forward to more in the future and please share and RT if you found this helpful & informative! We would love to tackle other topics that you request!
@CourtHarris
@Jaegermeister58
@ColinWestMDPhD
Thank you for joining us for our first brief #EBM #Tweetorial! Look forward to more in the future and please share and RT if you found this helpful & informative! We would love to tackle other topics that you request!
@CourtHarris
@Jaegermeister58
@ColinWestMDPhD
Tagging those who may enjoy this #EBM #MedEd content, but please share with others!
@AmyOxentenkoMD @EvidenceBasedMD @TheSGEM @VPrasadMDMPH @CStephenson_MD @BMJ_EBM @rbganatra @adamcifu @nataliexdean @medpedshosp @PaulBerglMD @RichardLehman1 @CarinaHimes @DKashiwagiMD
@AmyOxentenkoMD @EvidenceBasedMD @TheSGEM @VPrasadMDMPH @CStephenson_MD @BMJ_EBM @rbganatra @adamcifu @nataliexdean @medpedshosp @PaulBerglMD @RichardLehman1 @CarinaHimes @DKashiwagiMD
