Thread by @ADAlthousePhD

Noninferiority trials: a musing thread that I may regret.

NI trials are fickle & unsatisfying. Sometimes there's a legitimately good reason to do them (discussed below); the stats are maddening (also discussed below).

Suppose we have a particular group of patients that need to undergo a certain procedure. The procedure has a theoretical, biologically plausible risk of causing a particular complication; we generally give patients some prophylactic therapy against that complication.

Since no intervention is totally benign, we know that even this prophylactic therapy has its own risks, so we wonder if perhaps we could give patients a lower dose of it than typically used without losing the protective effect against the complication it's intended to prevent.

Perfectly sensible setting for a noninferiority trial: if the lower dose offers the same protection (or close!) as the higher dose, might be preferable, at least for some patients/clinical situations.

Of course, from here the statistical considerations get maddening. We have to decide on an acceptable NI margin, then we do this weird backwards test where we say we have significant evidence of noninferiority if the CI lies within that margin...makes everyone's head hurt.

(yes, you can argue that this would all feel a bit more coherent if we adopted a Bayesian approach & instead sought to accumulate a high probability that the true difference between groups was less than the NI margin; yay, no more "significance test" but still tricky)

Anyway, I thought of this recently because I saw an example of such a trial where some might argue that the noninferiority margin appears overly permissive, but I still think we're better off "having" this data to inform decisions than not having it.

You might be aware that appropriately "powering" an NI trial with a very small NI margin requires a huge sample size, often far larger than we can realistically accumulate (because it effectively means you need to get a very tight CI on "no effect" to prove NI with small margin)

So maybe we end up trials where the NI margin looks pretty big, and people are pretty unimpressed by their conclusions of noninferiority, arguing that the trial was basically a waste. I'm not sure that's true.

Suppose we end up seeing something **like** this.

Pre-spec NI margin of 4%. Trial recruits 1000 patients (500 per arm). 8 of 500 in high dose arm (1.6%) have bad outcome. 13 of 500 in low dose arm (2.6%) have bad outcome.

Absolute risk diff=1.0%, 95%CI from -0.8% to 2.8%; since it's within the pre-spec margin of 4% this trial concludes noninferiority for low-dose arm vs. high-dose arm, although the outcomes are slightly worse in the low-dose arm.

For an outcome that's so rare (1-2%) you might argue that a 4% noninferiority margin is unreasonable permissive. But powering the trial for a much smaller NI margin (1% or less) at this event rate jacks up the sample size you'd need.

If this hypothetical situation were indeed an open clinical question, e.g. if it were truly debated whether a lower dose of this prophylaxis was close enough to the higher dose to merit consideration, I'd argue that we're better off "having" this data than not having it at all.

Would the trialists love to just be able to magic up 10K patients instead of 1K for the trial? I'm sure they would, but if the choice is between "no data" and "1000 patient trial" aren't we better off having the small-ish trial to give folks some sense of just how big...

...the difference might actually be between a high dose vs. a low dose of prophylaxis in this setting?

Thanks for your attention. Have a nice day and please fly safely.

(mutes notifications, walks away)