Thread by @TenanATC

While I'm still bummed I can't be at #ACSM22 sharing work in person, here is a Thread about a new paper we published in @JOSPT titled:

"All MCIDs are Wrong, But Some May Be Useful"
jospt.org/doi/abs/10.251…

Buckle up!
1/17

It was great working with a bright, young PT on this paper @chrisB0Y3R, who brought a great clinical perspective.

This followed an earlier simulation published in @JAT_NATA showing that Minimal Clinically Important Difference metrics were biased



2/17

In this paper we used REAL DATA from Military Physical Therapy clinics to show empirical issues with MCIDs on multiple levels.

Remember if science is telling you there's "one cool trick" to say if your data are meaningful, there's always a catch (usually, a lot of them)

3/17

The basic MCID process is:
1. Take difference between pre and post scores
2. Determine your "anchor", which needs to be a binary outcome of interest (or an interval one you've dichtomotized [UGH])
3. Perform ROC Analysis...

4/17

4. Extract out change score which balances sensitivity and specificity (usually using Youden's J)

POOF! You have your magic number you want to achieve for "clinical relevance".

But wait...

5/17

You used a ROC, right? What was the Area Under the Curve? If your AUC is rubbish, then your MCID is rubbish.

Also, this is empirical data... right? Shouldn't there be confidence intervals around a point estimate? 🤔🤔

What about being biased by the baseline score?
6/17

Those 3 things are the entirety of this paper.

Basically, after you control for baseline score, you then need to have confidence intervals around BOTH your Area Under the Curve and the MCID to determine if you've got a valid MCID...

7/17

We concluded that if your AUC confidence intervals did not cross 0.5 (the noise line) then your MCID could potentially be statistically valid (e.g. better than noise)

PROMIS Pain Interference looks like this:

Small window in the upper-middle is potentially valid

8/17

After you look at AUC, you look at your MCID and those 95% confidence intervals. They're theoretically valid if the MCID and the intervals don't cross zero and are theoretically consistent.

Here's that image (explained in next tweet).

9/17

What do we mean by "theoretically consistent"?

Let's put it this way, would you really believe that a patient should have more pain interference in their life to reach a positive clinical outcome? Absolutely not, that's nonsense. That's all we're saying here.

10/17

So the AUC has to be better than noise and the MCID has to be theoretically consistent and those regions have to overlap. Where the MCID estimate is in BOTH of those windows, you've got a potentially valid MCID.

So you've got a conditional MCID...

11/17

You'd say something like, when your baseline score is between 50 and 75, we can calculate a valid MCID based on this graph. (see tweet number 9).

That's obviously a lot more complicated than a singular all-powerful MCID number most clinicians want.

12/17

Additionally, MCIDs have issues in that they treat clinical outcomes as univariate. Are you and your future life only dependent upon Pain Interference? Or Anxiety? or a Joint Range of Motion?

Certainly not. Humans are multidimensional and MCIDs don't capture that.

13/17

Moreover, many people get better or worse overtime. Clinical progression is anything but linear.

We can call these "state changes" and MCIDs don't really capture that dynamic and aren't designed to!

So what can we conclude? Well...

14/17

Current MCIDs are likely to be invalid, at least on some level based in either baseline bias, statistical validity, or theoretical consistency.

We've got a method to do MCIDs "better" but it's more complicated and still univariate.

15/17

I think there are a lot better ways to go about solving these problems.

First, model the multidimensional nature of patients with lots of data (multivariable analyses).

Second, look into state-transition models or other higher-level models that reflect reality...

16/17

So, yeah. We provocatively conclude that "the MCID may be too flawed a construct to accurately benchmark treatment outcomes".

The basic MCID may be simple, but it's wrong. There are better options, we just need to work towards them.

17/FIN