Thread by @dr_dmorgan

Do doctors understand how well treatments work?

We asked >500 clinicians in 8 US states

Chance that common treatments help an individual patient with
atrial fibrillation
hypertension
high cholesterol
osteoporosis

Open access: ja.ma/3A1u7N4 via @JAMANetwork

1/🧵

Recently we examined clinician understanding of probability in DIAGNOSIS

https://twitter.com/dr_dmorgan/status/1381667631099351040

The current question was similar but for TREATMENT

In the works, are numeracy, acceptance of uncertainty and other clinician personality factors associated with decisions?

2/🧵

Appreciate past interest from @tylercowen on probability in diagnosis.
In some ways, economists like him and @profemilyoster or statisticians like @natesilver538 have a better framework than doctors for assessing real life data, risk and tradeoffs

marginalrevolution.com/marginalrevolu…
3/🧵

SUMMARY 1/2
Clinicians tell patients they are likely to benefit from common treatments—which is not consistent with scientific evidence
Median clinician reported 30-50% chance of benefit
MUCH higher than the 1-3% chance from scientific evidence.
4/🧵

SUMMARY 2/2
Doctors who overestimated the most were most likely to use the treatment on real-life patients

If doctors and patients understood the true marginal effect of most treatments they likely would use less

5/🧵

How to think about risk:
Clinicians considering prescribing or patients considering taking a therapy need to know how likely a therapy is to benefit an individual patient

acpjournals.org/doi/full/10.73…
6/🧵

Patient benefit can be determined by:
1. estimating the risk of an adverse outcome of disease for an individual patient
2. apply the expected relative risk reduction (RRR) from the therapy
3. To identify the absolute reduction in risk for that patient (ARR)
7/🧵

While RRR is a property of a therapy across risk groups
ARR best reflects the potential benefit to an individual patient

Absolute numbers are recommended for physician and patient quantification of benefits of therapies
acpjournals.org/doi/10.7326/M1…

propublica.org/article/when-e…
8/🧵

The chance of benefit from a therapy,
along with chance of harm,
can then be used by clinicians and patients
when considering treatment decisions

acpjournals.org/doi/10.7326/M1…

ja.ma/3foo2SL
9/🧵

We are not the first to look at this but think our study most clearly explores and demonstrates the phenomenon of clinician overestimates of treatment benefits
(the consistency of findings is impressive)
10/🧵

A great review of heterogenous studies in @JAMAInternalMed
Concluded they underestimated harms and overestimated benefits
ja.ma/3xmjF0E
@PaulGlasziou
11/🧵

and an online survey of British GPs by @trishgreenhalgh found their knowledge of absolute harms and benefits were poor
bjgpopen.org/content/4/1/bj…

12/🧵

I summarized what was known before our study in @wapo for general public
@madden

https://twitter.com/dr_dmorgan/status/1347626857332670464

13/🧵

Methods:
Big team that asked many hard questions up front
@DKorenstein @AndrewFoy82 @masnick @SWeisenberg @ldscherer @LeykumLuci

We enrolled 542 doctors & NPs/PAs across 8 states w/ 76% response rate (response high mostly d/t in person enrollment)

14/🧵

We presented boringly simple cases of
atrial fibrillation
hypertension
osteoporosis
hypercholesterolemia

But all patients require nuance, so here is the survey

15/🧵

We found the best answers from scientific evidence w/ following approach and expert panel:
Given the message is challenging to clinicians (like me) we wanted to be extra transparent

16/🧵

Here’s what we found,
The clinician estimates of likelihood of disease outcomes and benefits of treatment were:
17/🧵

Warfarin for stroke prevention in atrial fibrillation
Patient with CHADS-VASC =1
~-.4-2% risk stroke 1 year
Clinician stroke risk estimates were ok, median 5%

chadsvasc.org

18/🧵

Expected patient benefits of anticoagulation were way too high in our scenario

“What would you tell Mr. Miller is the chance that warfarin will prevent him from having a stroke in the next year?”

0.2-1% scientific evidence
but clinicians say 0-100%, median 50%

19/🧵

And clinicians who estimate greater benefit of anticoagulation use it MUCH more often in real life (for the same patient!)

--varied care for the same patient ISN'T good

20/🧵

This matches studies finding frequent overuse of anticoagulation for low-risk atrial fibrillation patients
ja.ma/3xmQCKH
Where chance of major bleed exceeds stroke risk

21/🧵

We found similar findings for
hypertension
high cholesterol
osteoporosis

22/🧵

Mild hypertension and cardiovascular event
Clinician cardiovascular event risk estimates were accurate, median 10% (science 3-12%)
23/🧵

Expected patient benefits of antihypertensives were way too high
“What would you tell Mr. Davis is the chance that antihypertensive therapy will prevent him from having a cardiovascular event in the next 5 years?”
0-3% evidence
clinicians estimate 0-100%, median 30%

24/🧵

And clinicians who estimate greater benefit of antihypertensives use them more often in real life
But remarkable to me is how almost all would treat this patient

jamanetwork.com/journals/jama/…

25/🧵

Osteoporosis and hip fracture
Estimates of risk of hip fracture much higher than evidence, 10% vs. 1% or less

26/🧵

Expected patient benefits of bisphosphonates also way too high
“What would you tell Mr. Wilson is the chance that bisophosphonates will prevent her from having a hip fracture in the next 5 years?”
<1% scientific evidence
but clinicians estimate 0-100%, median 40%

27/🧵

And again, those who believe there is a greater chance of patient benefit are more likely to treat real life patients in their practice

28/🧵

Statins and cardiovascular events

Clinicians overestimated chance of cardiovascular event (or let’s be honest, gave same median 10% estimate as for other diseases)

29/🧵

Again, expected patient benefits of statins too high
“What would you tell Mr. Brown is the chance that moderate-intensity statin therapy will prevent him from having a cardiovascular event in the next year?”
0.3-2% scientific evidence
clinician estimate 0-100%, median 20%
30/🧵

And we see a great variation in clinician’s reporting they treat similar patients in their practice….

31/🧵

Interesting given statin guidance is closest to probability based thinking
tools.acc.org/ASCVD-Risk-Est…

@JeremySussman @ProfHayward
(I think statins are great, for the right patients)
32/🧵

Attendings were no better than residents (we don’t learn this after training)

33/🧵

We found clinicians do not consider the chance of a bad outcome and then estimate the proportion of that outcome that can be averted with treatments

Notably, Clinicians often gave same pretest estimate of risk for very different scenarios
34/🧵

Benefit was estimated at 20-50%
but in reality was 0.1-3%

Few clinicians used ARR for estimates of individual patient benefit although it is the best understood and most helpful metric of impact
acpjournals.org/doi/10.7326/00…

cmaj.ca/content/171/4/…
35/🧵

Why are estimates so far off?

Maybe doctors think RRR is how to think about treatments

Describing benefits in terms of RRR provides a much larger number than ARR and is associated with patients choosing therapy

link.springer.com/article/10.104…

36/🧵

RRR is often preferentially reported in the literature for:
research with small effect sizes

https://twitter.com/joshuadwallach/status/1330961697914429442?lang=en

@JoshuaDWallach
@jsross119

by the pharmaceutical industry
paho.org/hq/dmdocuments…
pubmed.ncbi.nlm.nih.gov/10349065/

37/🧵

practices that have been criticized
bmj.com/content/336/76…
38/🧵

Recent public misstatements regarding plasma therapy for COVID-19 illustrates how conflation of RRR and ARR can be misleading.
Interpreting an RRR of 35%, then @FDA commissioner stated

39/🧵

“35 out of 100 patients ‘would have been saved because of the administration of plasma.’”
nytimes.com/2020/08/24/hea…

However, the ARR was actually 3%

(and subsequent RCTs found NO benefit)
40/🧵

Clinicians do not receive extensive training or a consistent method to consider potential benefits and harms of therapy for individual patients,

41/🧵

Some approaches from yes/no–type guideline rec

or clinical experience, physiological thinking, or other theoretical models.

jclinepi.com/article/S0895-…

bedside-rounds.org/episode-50-i-k…

@AdamRodmanMD
@VPrasadMDMPH
42/🧵

Most respondents estimated the chance that the patient would benefit from a treatment to be a greater than the total chance that the same patient would have a negative outcome from disease.
(this is logically impossible)

43/🧵

In contrast to the clinical scenarios presented, in 1 hypothetical question, clinicians correctly calculated ARR from RRR.

Evidence-based principles of treatment may be understood theoretically but are not understood or applied to clinical care
bmcmededuc.biomedcentral.com/articles/10.11…
44/🧵

Overestimates of benefit are consistent with cognitive biases including
base rate neglect
anchoring bias, and
confirmation bias
10.1126/science.185.4157.1124

as well as the observation that humans often exaggerate small risks
doi.apa.org/doiLanding?doi…
45/🧵

Clinician estimates have been thought to be better for moderate risk conditions.

Our results argue more that there is a standard clinician estimate of risk that is relatively set regardless of condition or impact (10% outcome and 30-50% benefit Rx)
46 /🧵

and is not calibrated to allow for informed decisions or shared decisions with patients for low risk or benefit situations,

which are the majority encountered in outpatient medicine.
47 /🧵

Some would argue clinicians know but don’t explain to patients because it isn’t helpful
@anish_koka
But our results suggest they don’t think of it that way

I believe us doctors know medical social norms, how we act
48/🧵

—we treat high bp or afib, and, if asked, can estimate numbers that would support that

--the probability estimates come after the feeling or decision to treat
49/🧵

This has big implications for #SDM:
Clinician overestimates of treatment benefits would severely limit shared decision-making

acpjournals.org/doi/10.7326/M1…
50/🧵

Also care generally
Bias is consistent across doctors, hospitals, pharmacists, Pharma, researchers and patients.

We expect that treatments mostly help patients

When research shows most interventions have marginal benefits and are given for a chance of benefit.

51/🧵

How we could do better:

@richardlehman described these estimates as similar to using GPS or satnav—a useful guide to getting around that can be improved with expert knowledge

Why wouldn’t clinicians want that?

go.shr.lc/2UMgv9y
52/🧵

If clinicians can better understand expected effects of common therapy and convey them to patients across the tens- of thousands of interactions daily, we could make medicine more scientifically based and high-value
53/🧵
@drjohnm

Thanks for sticking with me
I hope this was stimulating
Our next analysis will look at clinician personality, numeracy and other individual factors to explain variation in use of testing
54/🧵

Needless to say, this is a lot of thinking inspired by 'medical conservatives' like @drjohnm @adamcifu @AndrewFoy82 @VPrasadMDMPH

Sorry, I know tweetorials are supposed to be shorter than the article!