The current question was similar but for TREATMENT
In the works, are numeracy, acceptance of uncertainty and other clinician personality factors associated with decisions?
2/🧵
Appreciate past interest from @tylercowen on probability in diagnosis.
In some ways, economists like him and @profemilyoster or statisticians like @natesilver538 have a better framework than doctors for assessing real life data, risk and tradeoffs
SUMMARY 1/2
Clinicians tell patients they are likely to benefit from common treatments—which is not consistent with scientific evidence
Median clinician reported 30-50% chance of benefit
MUCH higher than the 1-3% chance from scientific evidence.
4/🧵
SUMMARY 2/2
Doctors who overestimated the most were most likely to use the treatment on real-life patients
If doctors and patients understood the true marginal effect of most treatments they likely would use less
5/🧵
How to think about risk:
Clinicians considering prescribing or patients considering taking a therapy need to know how likely a therapy is to benefit an individual patient
Patient benefit can be determined by: 1. estimating the risk of an adverse outcome of disease for an individual patient 2. apply the expected relative risk reduction (RRR) from the therapy 3. To identify the absolute reduction in risk for that patient (ARR)
7/🧵
While RRR is a property of a therapy across risk groups
ARR best reflects the potential benefit to an individual patient
Absolute numbers are recommended for physician and patient quantification of benefits of therapies acpjournals.org/doi/10.7326/M1…
We are not the first to look at this but think our study most clearly explores and demonstrates the phenomenon of clinician overestimates of treatment benefits
(the consistency of findings is impressive)
10/🧵
We enrolled 542 doctors & NPs/PAs across 8 states w/ 76% response rate (response high mostly d/t in person enrollment)
14/🧵
We presented boringly simple cases of
atrial fibrillation
hypertension
osteoporosis
hypercholesterolemia
But all patients require nuance, so here is the survey
15/🧵
We found the best answers from scientific evidence w/ following approach and expert panel:
Given the message is challenging to clinicians (like me) we wanted to be extra transparent
16/🧵
Here’s what we found,
The clinician estimates of likelihood of disease outcomes and benefits of treatment were:
17/🧵
Warfarin for stroke prevention in atrial fibrillation
Patient with CHADS-VASC =1
~-.4-2% risk stroke 1 year
Clinician stroke risk estimates were ok, median 5%
Expected patient benefits of anticoagulation were way too high in our scenario
“What would you tell Mr. Miller is the chance that warfarin will prevent him from having a stroke in the next year?”
0.2-1% scientific evidence
but clinicians say 0-100%, median 50%
19/🧵
And clinicians who estimate greater benefit of anticoagulation use it MUCH more often in real life (for the same patient!)
--varied care for the same patient ISN'T good
20/🧵
This matches studies finding frequent overuse of anticoagulation for low-risk atrial fibrillation patients ja.ma/3xmQCKH
Where chance of major bleed exceeds stroke risk
21/🧵
We found similar findings for
hypertension
high cholesterol
osteoporosis
22/🧵
Mild hypertension and cardiovascular event
Clinician cardiovascular event risk estimates were accurate, median 10% (science 3-12%)
23/🧵
Expected patient benefits of antihypertensives were way too high
“What would you tell Mr. Davis is the chance that antihypertensive therapy will prevent him from having a cardiovascular event in the next 5 years?”
0-3% evidence
clinicians estimate 0-100%, median 30%
24/🧵
And clinicians who estimate greater benefit of antihypertensives use them more often in real life
But remarkable to me is how almost all would treat this patient
Osteoporosis and hip fracture
Estimates of risk of hip fracture much higher than evidence, 10% vs. 1% or less
26/🧵
Expected patient benefits of bisphosphonates also way too high
“What would you tell Mr. Wilson is the chance that bisophosphonates will prevent her from having a hip fracture in the next 5 years?”
<1% scientific evidence
but clinicians estimate 0-100%, median 40%
27/🧵
And again, those who believe there is a greater chance of patient benefit are more likely to treat real life patients in their practice
28/🧵
Statins and cardiovascular events
Clinicians overestimated chance of cardiovascular event (or let’s be honest, gave same median 10% estimate as for other diseases)
29/🧵
Again, expected patient benefits of statins too high
“What would you tell Mr. Brown is the chance that moderate-intensity statin therapy will prevent him from having a cardiovascular event in the next year?”
0.3-2% scientific evidence
clinician estimate 0-100%, median 20%
30/🧵
And we see a great variation in clinician’s reporting they treat similar patients in their practice….
Attendings were no better than residents (we don’t learn this after training)
33/🧵
We found clinicians do not consider the chance of a bad outcome and then estimate the proportion of that outcome that can be averted with treatments
Notably, Clinicians often gave same pretest estimate of risk for very different scenarios
34/🧵
Benefit was estimated at 20-50%
but in reality was 0.1-3%
Few clinicians used ARR for estimates of individual patient benefit although it is the best understood and most helpful metric of impact acpjournals.org/doi/10.7326/00…
Recent public misstatements regarding plasma therapy for COVID-19 illustrates how conflation of RRR and ARR can be misleading.
Interpreting an RRR of 35%, then @FDA commissioner stated
39/🧵
“35 out of 100 patients ‘would have been saved because of the administration of plasma.’” nytimes.com/2020/08/24/hea…
However, the ARR was actually 3%
(and subsequent RCTs found NO benefit)
40/🧵
Clinicians do not receive extensive training or a consistent method to consider potential benefits and harms of therapy for individual patients,
41/🧵
Some approaches from yes/no–type guideline rec
or clinical experience, physiological thinking, or other theoretical models.
Most respondents estimated the chance that the patient would benefit from a treatment to be a greater than the total chance that the same patient would have a negative outcome from disease.
(this is logically impossible)
43/🧵
In contrast to the clinical scenarios presented, in 1 hypothetical question, clinicians correctly calculated ARR from RRR.
Overestimates of benefit are consistent with cognitive biases including
base rate neglect
anchoring bias, and
confirmation bias
10.1126/science.185.4157.1124
Clinician estimates have been thought to be better for moderate risk conditions.
Our results argue more that there is a standard clinician estimate of risk that is relatively set regardless of condition or impact (10% outcome and 30-50% benefit Rx)
46 /🧵
and is not calibrated to allow for informed decisions or shared decisions with patients for low risk or benefit situations,
which are the majority encountered in outpatient medicine.
47 /🧵
Some would argue clinicians know but don’t explain to patients because it isn’t helpful @anish_koka
But our results suggest they don’t think of it that way
I believe us doctors know medical social norms, how we act
48/🧵
—we treat high bp or afib, and, if asked, can estimate numbers that would support that
--the probability estimates come after the feeling or decision to treat
49/🧵
This has big implications for #SDM:
Clinician overestimates of treatment benefits would severely limit shared decision-making
If clinicians can better understand expected effects of common therapy and convey them to patients across the tens- of thousands of interactions daily, we could make medicine more scientifically based and high-value
53/🧵 @drjohnm
Thanks for sticking with me
I hope this was stimulating
Our next analysis will look at clinician personality, numeracy and other individual factors to explain variation in use of testing
54/🧵
This is the most relevant paper I have written. Not perfect but addresses a huge issue I think could change medicine if acknowledged
…it has changed how I think about diagnosis
summary:
Clinicians widely overestimated chance of disease especially after testing
Cardiac ischemia after + ECG—EBM 2-11%, median answer 70%
UTI after + urine cx—EBM 0-8.3%, answer 80%
Breast CA after + mammo—EBM 3-9%, answer 50%
Pneumonia after + CXR EBM 46-65%, answer 95%
Gerd Gigerenzer, David Eddy, @StevenWoloshin@arjunmanrai & others asked how well doctors do at the math of understanding diagnosis, and found they aren’t great.
The @US_FDA has a test comparison site that is incomprehensible to me… but @ASMicrobiology types tell me it reports on analytical sensitivity and LoD for tests
3/n
Clearly, the words physicians use have
a critical function in this communication
Referring to harms as “risks” emphasizes that
the unfavorable outcome may or may not happen,
whereas there is no parallel language that highlights
the equally probabilistic nature of “benefits.”