Thread by @raj_mehta

1/ Thread on Analyzing Physician Thought Processes with Conditional Probability

One important aspect of #EBM is to help physicians calibrate their probabilistic reasoning and improve their clinical judgement.

We will walk through one example from study highlighted below.

2/ The Study:

The basis of this thread is this interesting paper on physician probability estimates of medical outcomes:

jamanetwork.com/journals/jaman…

3/ Key concepts:

Difference between Independent vs dependent events
en.wikipedia.org/wiki/Independe…

Conditional Probability
en.wikipedia.org/wiki/Condition…

Summary of Probabilities (picture)

4/ Case Scenario:

We are presented with a patient with a pulmonary nodule.

Assuming our first decision point is deciding between surveillance or biopsy, we need to start by estimating the probability of cancer.

5/ Reading a Case:

When Docs read:
"What is the probability that a Patient with B,C,D... has cancer?"

We interpret "B,C,D" in the past tense as GIVEN information.

Thus, we estimate chance of cancer GIVEN age, smoking history, 2.5 cm nodule, etc

6/ Estimating Probability of Cancer:

Using a Risk Tool, we get risk ~ 80%

This is consistent with most surveyed physicians from the study (Component 1 below, majority >75%).

It suggests physician judgement is pretty good at incorporating conditional probability in estimation!

7/ A tricky question:

How do we interpret the question in Frame 1 below?

"What is the overall combined probability that (1)... and (2)... ?"

(1) the patient has malignancy
(2) a technically successful transthoracic needle biopsy will reveal malignancy on pathological exam

8/ How different are the following examples?

1. "What is the overall combined probability that the patient with malignancy has a biopsy that will reveal cancer"

2. "What is the overall combined probability that the patient has malignancy and a biopsy will reveal cancer"

9/ When we ask physicians to "combine probabilities", what is their cognitive process?

The authors intended to ask:
"what is joint probability that (1) & (2) occur independently?"

But the question could also be read as:
"what is the probability of (2) given (1)?"

10/ The ambiguity of the question in Frame 1 is a major limitation to this study.

The high rate of physician "fallacy" may just be due to differing interpretations of the case question, rather than any failure in multi-step (or conditional) probability estimation.

11/ The paper does highlight an important aspect of physician judgement that is an area for improvement:

Improving calibration of our estimates

Take the example below. Although most surveyed docs (component 1) estimated a high risk of cancer, a minority did not (~ 7 dots <50%)

12/ Many of those physicians might correctly think that cancer is the most likely diagnosis (we are good at discriminating signs/symptoms/data to determine the most likely outcome).

But experience does not always help us calibrate the most accurate probability estimates.

13/ EBM serves as an important tool, because physicians can compare the cognitive probability estimates against validated evidence (epidemiologic data, risk tools, etc).

This can help us refine and improve the areas where our estimates are inaccurate.

14/ The cognitive process in physician judgement involves a complex array of conditional probability estimates. Modeling these steps can be tricky.

Most physicians likely do a reasonably good job, but even best expert can still find opportunities for improvement.

/End

Addendum:

Good paper on some limits of using probability to model cognitive process, & how subjective uncertainty and surprise helps us to learn from event outcomes.

frontiersin.org/articles/10.33…