Do doctors understand how well treatments work?

We asked >500 clinicians in 8 US states

Chance that common treatments help an individual patient with
atrial fibrillation
hypertension
high cholesterol
osteoporosis

Open access: ja.ma/3A1u7N4 via @JAMANetwork

1/🧵
Recently we examined clinician understanding of probability in DIAGNOSIS

The current question was similar but for TREATMENT

In the works, are numeracy, acceptance of uncertainty and other clinician personality factors associated with decisions?

2/🧵
Appreciate past interest from @tylercowen on probability in diagnosis.
In some ways, economists like him and @profemilyoster or statisticians like @natesilver538 have a better framework than doctors for assessing real life data, risk and tradeoffs

marginalrevolution.com/marginalrevolu…
3/🧵
SUMMARY 1/2
Clinicians tell patients they are likely to benefit from common treatments—which is not consistent with scientific evidence
Median clinician reported 30-50% chance of benefit
MUCH higher than the 1-3% chance from scientific evidence.
4/🧵
SUMMARY 2/2
Doctors who overestimated the most were most likely to use the treatment on real-life patients

If doctors and patients understood the true marginal effect of most treatments they likely would use less

5/🧵
How to think about risk:
Clinicians considering prescribing or patients considering taking a therapy need to know how likely a therapy is to benefit an individual patient

acpjournals.org/doi/full/10.73…
6/🧵
Patient benefit can be determined by:
1. estimating the risk of an adverse outcome of disease for an individual patient
2. apply the expected relative risk reduction (RRR) from the therapy
3. To identify the absolute reduction in risk for that patient (ARR)
7/🧵
While RRR is a property of a therapy across risk groups
ARR best reflects the potential benefit to an individual patient

Absolute numbers are recommended for physician and patient quantification of benefits of therapies
acpjournals.org/doi/10.7326/M1…

propublica.org/article/when-e…
8/🧵
The chance of benefit from a therapy,
along with chance of harm,
can then be used by clinicians and patients
when considering treatment decisions

acpjournals.org/doi/10.7326/M1…

ja.ma/3foo2SL
9/🧵
We are not the first to look at this but think our study most clearly explores and demonstrates the phenomenon of clinician overestimates of treatment benefits
(the consistency of findings is impressive)
10/🧵
A great review of heterogenous studies in @JAMAInternalMed
Concluded they underestimated harms and overestimated benefits
ja.ma/3xmjF0E
@PaulGlasziou
11/🧵
and an online survey of British GPs by @trishgreenhalgh found their knowledge of absolute harms and benefits were poor
bjgpopen.org/content/4/1/bj…

12/🧵
I summarized what was known before our study in @wapo for general public
@madden



13/🧵
Methods:
Big team that asked many hard questions up front
@DKorenstein @AndrewFoy82 @masnick @SWeisenberg @ldscherer @LeykumLuci

We enrolled 542 doctors & NPs/PAs across 8 states w/ 76% response rate (response high mostly d/t in person enrollment)

14/🧵
We presented boringly simple cases of
atrial fibrillation
hypertension
osteoporosis
hypercholesterolemia

But all patients require nuance, so here is the survey

15/🧵
We found the best answers from scientific evidence w/ following approach and expert panel:
Given the message is challenging to clinicians (like me) we wanted to be extra transparent

16/🧵
Here’s what we found,
The clinician estimates of likelihood of disease outcomes and benefits of treatment were:
17/🧵
Warfarin for stroke prevention in atrial fibrillation
Patient with CHADS-VASC =1
~-.4-2% risk stroke 1 year
Clinician stroke risk estimates were ok, median 5%

chadsvasc.org

18/🧵
Expected patient benefits of anticoagulation were way too high in our scenario

“What would you tell Mr. Miller is the chance that warfarin will prevent him from having a stroke in the next year?”

0.2-1% scientific evidence
but clinicians say 0-100%, median 50%

19/🧵
And clinicians who estimate greater benefit of anticoagulation use it MUCH more often in real life (for the same patient!)

--varied care for the same patient ISN'T good

20/🧵
This matches studies finding frequent overuse of anticoagulation for low-risk atrial fibrillation patients
ja.ma/3xmQCKH
Where chance of major bleed exceeds stroke risk

21/🧵
We found similar findings for
hypertension
high cholesterol
osteoporosis

22/🧵
Mild hypertension and cardiovascular event
Clinician cardiovascular event risk estimates were accurate, median 10% (science 3-12%)
23/🧵
Expected patient benefits of antihypertensives were way too high
“What would you tell Mr. Davis is the chance that antihypertensive therapy will prevent him from having a cardiovascular event in the next 5 years?”
0-3% evidence
clinicians estimate 0-100%, median 30%

24/🧵
And clinicians who estimate greater benefit of antihypertensives use them more often in real life
But remarkable to me is how almost all would treat this patient

jamanetwork.com/journals/jama/…

25/🧵
Osteoporosis and hip fracture
Estimates of risk of hip fracture much higher than evidence, 10% vs. 1% or less

26/🧵
Expected patient benefits of bisphosphonates also way too high
“What would you tell Mr. Wilson is the chance that bisophosphonates will prevent her from having a hip fracture in the next 5 years?”
<1% scientific evidence
but clinicians estimate 0-100%, median 40%

27/🧵
And again, those who believe there is a greater chance of patient benefit are more likely to treat real life patients in their practice

28/🧵
Statins and cardiovascular events

Clinicians overestimated chance of cardiovascular event (or let’s be honest, gave same median 10% estimate as for other diseases)

29/🧵
Again, expected patient benefits of statins too high
“What would you tell Mr. Brown is the chance that moderate-intensity statin therapy will prevent him from having a cardiovascular event in the next year?”
0.3-2% scientific evidence
clinician estimate 0-100%, median 20%
30/🧵
And we see a great variation in clinician’s reporting they treat similar patients in their practice….

31/🧵
Interesting given statin guidance is closest to probability based thinking
tools.acc.org/ASCVD-Risk-Est…

@JeremySussman @ProfHayward
(I think statins are great, for the right patients)
32/🧵
Attendings were no better than residents (we don’t learn this after training)

33/🧵
We found clinicians do not consider the chance of a bad outcome and then estimate the proportion of that outcome that can be averted with treatments

Notably, Clinicians often gave same pretest estimate of risk for very different scenarios
34/🧵
Benefit was estimated at 20-50%
but in reality was 0.1-3%

Few clinicians used ARR for estimates of individual patient benefit although it is the best understood and most helpful metric of impact
acpjournals.org/doi/10.7326/00…

cmaj.ca/content/171/4/…
35/🧵
Why are estimates so far off?

Maybe doctors think RRR is how to think about treatments

Describing benefits in terms of RRR provides a much larger number than ARR and is associated with patients choosing therapy

link.springer.com/article/10.104…

36/🧵
RRR is often preferentially reported in the literature for:
research with small effect sizes


@JoshuaDWallach
@jsross119

by the pharmaceutical industry
paho.org/hq/dmdocuments…
pubmed.ncbi.nlm.nih.gov/10349065/

37/🧵
practices that have been criticized
bmj.com/content/336/76…
38/🧵
Recent public misstatements regarding plasma therapy for COVID-19 illustrates how conflation of RRR and ARR can be misleading.
Interpreting an RRR of 35%, then @FDA commissioner stated

39/🧵
“35 out of 100 patients ‘would have been saved because of the administration of plasma.’”
nytimes.com/2020/08/24/hea…

However, the ARR was actually 3%

(and subsequent RCTs found NO benefit)
40/🧵
Clinicians do not receive extensive training or a consistent method to consider potential benefits and harms of therapy for individual patients,

41/🧵
Some approaches from yes/no–type guideline rec

or clinical experience, physiological thinking, or other theoretical models.

jclinepi.com/article/S0895-…

bedside-rounds.org/episode-50-i-k…

@AdamRodmanMD
@VPrasadMDMPH
42/🧵
Most respondents estimated the chance that the patient would benefit from a treatment to be a greater than the total chance that the same patient would have a negative outcome from disease.
(this is logically impossible)

43/🧵
In contrast to the clinical scenarios presented, in 1 hypothetical question, clinicians correctly calculated ARR from RRR.

Evidence-based principles of treatment may be understood theoretically but are not understood or applied to clinical care
bmcmededuc.biomedcentral.com/articles/10.11…
44/🧵
Overestimates of benefit are consistent with cognitive biases including
base rate neglect
anchoring bias, and
confirmation bias
10.1126/science.185.4157.1124

as well as the observation that humans often exaggerate small risks
doi.apa.org/doiLanding?doi…
45/🧵
Clinician estimates have been thought to be better for moderate risk conditions.

Our results argue more that there is a standard clinician estimate of risk that is relatively set regardless of condition or impact (10% outcome and 30-50% benefit Rx)
46 /🧵
and is not calibrated to allow for informed decisions or shared decisions with patients for low risk or benefit situations,

which are the majority encountered in outpatient medicine.
47 /🧵
Some would argue clinicians know but don’t explain to patients because it isn’t helpful
@anish_koka
But our results suggest they don’t think of it that way

I believe us doctors know medical social norms, how we act
48/🧵
—we treat high bp or afib, and, if asked, can estimate numbers that would support that

--the probability estimates come after the feeling or decision to treat
49/🧵
This has big implications for #SDM:
Clinician overestimates of treatment benefits would severely limit shared decision-making

acpjournals.org/doi/10.7326/M1…
50/🧵
Also care generally
Bias is consistent across doctors, hospitals, pharmacists, Pharma, researchers and patients.

We expect that treatments mostly help patients

When research shows most interventions have marginal benefits and are given for a chance of benefit.

51/🧵
How we could do better:

@richardlehman described these estimates as similar to using GPS or satnav—a useful guide to getting around that can be improved with expert knowledge

Why wouldn’t clinicians want that?

go.shr.lc/2UMgv9y
52/🧵
If clinicians can better understand expected effects of common therapy and convey them to patients across the tens- of thousands of interactions daily, we could make medicine more scientifically based and high-value
53/🧵
@drjohnm
Thanks for sticking with me
I hope this was stimulating
Our next analysis will look at clinician personality, numeracy and other individual factors to explain variation in use of testing
54/🧵
Needless to say, this is a lot of thinking inspired by 'medical conservatives' like @drjohnm @adamcifu @AndrewFoy82 @VPrasadMDMPH
Sorry, I know tweetorials are supposed to be shorter than the article!

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Dan Morgan

Dan Morgan Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @dr_dmorgan

12 Apr
How good are doctors at diagnosis?

This is the most relevant paper I have written. Not perfect but addresses a huge issue I think could change medicine if acknowledged
…it has changed how I think about diagnosis

ja.ma/3rQNtjv
@JAMAInternalMed
@drjohnm
1/n
summary:
Clinicians widely overestimated chance of disease especially after testing

Cardiac ischemia after + ECG—EBM 2-11%, median answer 70%
UTI after + urine cx—EBM 0-8.3%, answer 80%
Breast CA after + mammo—EBM 3-9%, answer 50%
Pneumonia after + CXR EBM 46-65%, answer 95%
Gerd Gigerenzer, David Eddy, @StevenWoloshin @arjunmanrai & others asked how well doctors do at the math of understanding diagnosis, and found they aren’t great.

Many of issues w/ real life tests covered by @deeksj @d_spiegel @dan_diekema
Read 20 tweets
8 Jan
Like @BenMazer, I was bummed to have a op-ed during the riot

In it, I describe the reality that most medical treatments have very marginal effects.

A reality with big implications

d/w @VPrasadMDMPH @lowninstitute @AndrewFoy82 @drjohnm

washingtonpost.com/outlook/2021/0…
1/
2/ If clinicians understood small chance of benefit with most Rx, I feel we would make very different decisions for most patients

Thanks @MikeMadden editor

By taking a medication daily we buy a lottery ticket with a payout often as low as 1% or 2%.
3/ And we can’t know if we won or not…as winning usually means "nothing" happens.

Of course, we assume no bad outcome = benefits,

so 95% or more appear to benefit, not the 1-2% who truly benefit from RCTs
Read 10 tweets
21 Dec 20
We set out to identify the sensitivity & specificity of common tests for #COVID19 along w/ @dan_diekema , @Anthony98947615 w/ @CDCgov support

A simple enough task, right?

I’ve seen tweets by @DrSidMukherjee @drjohnm @BenMazer @PaulSaxMD @BradSpellberg and others interested
1/n
Looking for comments/criticism

What are we missing? No industry adverts please!

Important papers?
2/n
The @US_FDA has a test comparison site that is incomprehensible to me… but @ASMicrobiology types tell me it reports on analytical sensitivity and LoD for tests
3/n
Read 20 tweets
15 Sep 20
Catch yourself when you say “risks vs. benefits” because you aren’t making a fair comparison.

In @JAMA_current ja.ma/336Lj4Y
& podcast ja.ma/2FAGKI2

@eliowa @drjohnm @d_spiegel @zeynep @VPrasadMDMPH

Why we say it and what we think is better below...
This building block of clinical decisions biases by framing uncertain harm vs. certain benefits and nudges towards treatment

Written with
@DKorenstein @ldscherer
(Over 2 years, i'm embarrassed to admit)
Clearly, the words physicians use have
a critical function in this communication

Referring to harms as “risks” emphasizes that
the unfavorable outcome may or may not happen,
whereas there is no parallel language that highlights
the equally probabilistic nature of “benefits.”
Read 8 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(