Thread by @venkmurthy

Recently the CONSERVE trial was cited on twitter as supporting use of CCTA b/c it showed that patients sent for ICA could undergo CCTA instead w/o an increase in events.

imaging.onlinejacc.org/content/12/7_P…

This is a non-inferiority trial but I can't make the math add up.

Follow along...

In a non-inferiority trial, we are testing whether ONE bound of the confidence interval for the primary endpoint crosses a non-inferiority margin.

Usually this is upper bound for MACE in cardiology trials showing new tx doesn't result in significantly more MACE than old tx.

There is much controversy over how much worse a tx can be before we think it is close enough.

Some trials have tested upper bounds of non-inferiority of as high 1.5 or even higher maybe.

In CONSERVE the tested upper bound was 1.33.

This means CCTA patients could experience 33% more MACE events than ICA and still be non-inferior.

Let's put this another way: if a new therapy decreased MACE events in patients referred to ICA by 25% would you want it?

I would want it to. My sense is that non-inferiority margins of 1.33 for MACE are too high.

Why set them this high? Because if you don't the sample sizes get VERY large very quickly.

In this trial, the point estimate for MACE was 0.99 which is damn close to 1.0. I'm happy about that.

But the 95% confidence interval for that hazard ratio is 0.66-1.47.

Interesting. Is 1.47>1.33?

So how did the authors conclude the trial made it's non-inferiority endpoint?

Simple: they didn't ask whether the 95% CI crossed 1.33, they asked whether the 90% CI crossed 1.33

(in other words a ONE sided p-value of 0.05 or alpha of 0.05 rather than a TWO sided p-value of 0.05 or alpha of 0.025)

They don't mention that in the results paper, but they do mention it in the design paper.

sciencedirect.com/science/articl…

Is this okay? Nearly everyone I know thinks that it is stretching the limits. The FDA guidance certainly seems to set an expectation that the 95% confidence interval (i.e. alpha of 0.025)

fda.gov/media/78504/do…

But even then, the numbers don't seem to add up for me.

Let's do the math, shall we?

In CONSERVE. The HR is reported as 0.99 [95% CI 0.66-1.47].

To calculate the upper 90% confidence interval, we need for first take the natural log of the HR and the upper and lower confidence intervals.

Let me do this quickly in Excel (shame on me for using Excel!)

The natural log of the hazard ratio is called beta. It is what is estimated in many types of survival regression including the Cox proportional hazards regression used here.

The upper 95% confidence interval for beta is given by taking the estimate for beta and adding 1.96 times its standard error.

Likewise the lower 95% confidence interval for is given by taking the estimate for beta and subtracting 1.96 times it standard error.

In other words, you can get the standard error by taking the DIFFERENCE between the upper and lower confidence intervals for beta and dividing by 1.96*2 = 3.92

So if we take the difference BETWEEN 0.385 and -0.416 we get 0.801.

Dividing that by 2.93 gives us 0.204.

Take a breath!

So we now know beta is -0.010 and has a standard error of 0.204

To get the upper 90% confidence interval for beta, you take beta and add 1.645 TIMES the standard error.

-0.010 + 1.645 * 0.204 = 0.326

Now, to convert the upper 90% confidence interval for beta to an upper 90% confidence interval for the hazard ratio, you take the exponent of it.

e^0.326 = 1.39

That's pretty close to 1.33 but still crosses it.

So what do we know?

* Very high non-inferiority margin
* 95% CI crosses the margin
* 90% CI crosses the margin

But somehow it met its non-inferiority endpoint.

End of the day, the point estimate is very close to 1.0 so maybe it doesn't matter. But I think it does. The investigators picked a VERY lax margin and appeared to miss it anyway but it is still reported as making it.

I think that it is important.

Do you?

Was this helpful?

This is an interesting comment. Not 100% sure what to make of it. There are other things that I'm confused byin this report, but I'm not ready to say more.

What's above shows the trial did NOT make its primary endpoint, despite the claim that it did

https://twitter.com/Katkatakat1/status/1160257670831583234?s=20

It matters if a trial has basic math errors in the primary endpoint.

They did not demonstrate non-inferiority. They happened to have HR of 1.0 but the trial is compatible with a markedly worse event rate.

https://twitter.com/onco_cardiology/status/1160357259911864321?s=19

Thread continues here. Please click this:

https://twitter.com/onco_cardiology/status/1160359809474080769?s=19

3.92! Typo in the tweet not in the result