Thread by @RitaINdeFaria

I'm now live tweeting from the #ISPOREurope session Much Ado About Little: Dealing with Limited RCT Evidence for Early HTA and Reimbursement Decisions with @MJSculpher, @SBujkiewicz, Eva Dietrich, Steven Palmer from @CHEyork, and @UweSiebert9

@MJSculpher starts by setting the context and introducing the speakers.
@UweSiebert9 is the 1st plenary speaker, on causal inference in #RCTs and #RWE
#ISPOREurope

When do we need causal inference methoss? when there's no randomisation; or the randomisation was broken (e.g. treatment switches)
Throughout his talk, Uwe will use the example of 2nd line treatment in women with ovarian cancer who progressed
#ISPOREurope

Target trial concept: the idea is to mimic a hypothetical RCT using #RWD. Time zero is when the trial starts.
e.g. eligibility (=Population).

#ISPOREurope

E.g. treatment strategies. Often in #RWD studies, studies compare ever treated vs never treated. There can be immortal time bias, as people who live longer are more likely to be ever treated.

We should compare well defined treatment strategies

#ISPOREurope

Causal graphs are essential tool for causal inference. Useful to identify and handle time-dependent confounding.

As treatment influences the severity of disease, traditional adjustment methods fail

#ISPOREurope

We should use causal inference methods, such as g-formula, marginal structural models with inverse probability weighting, and structural nested models with g-estimation.

#ISPOREurope

In this example, we can compare our #RWE analysis with the original RCT, and investigate the bias depending on the methods.
All traditional methods led to biased results apart from the target trial approach + g-methods

#ISPOREurope

There are applications in the context of limited trial evidence.
Causal methods with or without the target trial approach (depending if there is a RCT) can be applied to inform #HTA and #CostEffectiveness

#ISPOREurope

Next up, @SBujkiewicz on surrogate endpoints in #HTA decision making.

Modern trials of novel therapies bring various challenges due to small subsets of pop, reduced mortality rates requiring long-followup --> treatment effect on OS can have large uncertainty

#ISPOREurope

Regulators can carry out conditional licensing based on TE on surrogate endpoints such as PFS --> implications for HTA as the outcomes of interest often include OS

Surrogate endpoints need to be validated.

#ISPOREurope

Sylwia will focus today Bayesian evidence synthesis to validate surrogate endpoint validation, referring the @NICE_DSU TSD20
these methods take account of both correlation and uncertainty
#ISPOREurope

But a number of challenges remain, as there may be a small number of RCTs or their sample size may be small.

A new paper allows to model the surrogate relationships in greater detail
#ISPOREurope

this method was illustrated in advanced colorectal cancer, for tumour response as a surrogate to PFS.
The correlation was -0.67 with moderate uncertainty with pairwise MA. With the new method, the correlation was greater and the predictions were improved.

#ISPOREurope

Recent development, using hierarchical methods, also applied to adv colorectal cancer.

They represented the associations between PFS and OS using the credible intervals from the slopes. This method was able to reduce uncertainty

#ISPOREurope

What about using #RWE for surrogate endpoint evaluation? RWE can offer longer follow-up and larger sample size.

Paper in development about combining data from #RCT and cohort studies

#ISPOREurope

The uncertainty around the correlation reduced and they were able to achieve more precise predictions.

This method could also be used to combine single arm trial with other RCTs

#ISPOREurope

If we want to use putative surrogate endpoints, we should evaluate them and consider uncertainty.

The strength of the surrogate relationship drives the level of uncertainty around the predicted treatment effect on final outcome

#ISPOREurope

Next up at #ISPOREurope is Steve Palmer from @CHEyork, on modelling approaches for the evaluation of histology independent treatments

Basket trials have been proposed to assess multiple drugs and/or populations tested in parallel. The patients have a common genomic alteration regardless of the histological type of tumour. While there are clear efficiencies, they present challenges for HTA
#ISPOREurope

To date, 3 histology independent products have been approved by the FDA. All have been approved based on phase 1/phase 2 trials, with 54-149 patients depending on product.

Example of the type of data available with larotrectinib.

#ISPOREurope

The heterogeneity of the effect across the different subpopulations is a key issue. They could be independently separately, but we will lose efficiency. If we pool all patients, then we are ignoring heterogeneity and assuming equal efficacy.

#ISPOREurope

Another approach are Bayesian hierarchical models, because they allow for borrowing information across tumour subtypes and accounts for heterogeneity in the probability of response, and allows to predict response in unrepresented tumours

#ISPOREurope

Here are some results: the ORR is smaller and more uncertain if we use the BHM compared to pooling all populations.
Recently, more evidence has emerged, and the BHM was updated accordingly, and the ORR predicted by the BHM was closer to the pooled ORR.

#ISPOREurope

What are the implications for #HTA?
Even if we have homogeneity in response, it may not translate into homogeneity in policy relevant outcomes, given the different natural history, cost of detection and comparator costs

#ISPOREurope

Steve presents a case study based on a hypothetical drug. They explored different approval policies and alternative decision metrics, such as population net health benefit, consequences of decision uncertainty & of heterogeneity
#ISPOREurope

With these methods, they showed the health consequences of uncertainty at the per person and at the pop level. These methods help inform the value of further evidence collection and to inform stratified decisions.

#ISPOREurope

In conclusion, these new regulatory pathways bring opportunities and obstacles. We need to think to move beyond the ICER, to think about the impact on pop health, consequences of uncertainty & heterogeneity, and ways to manage risk

#ISPOREurope

We now have Eva Dietrich from the University of Bonn Pharmaceutical Institute.

Eva notes that, in the mid-90s, the FDA approved most drugs based on at least 2 pivotal trials, but has changed. Do these drugs fulfill these high expectations?

#ISPOREurope

Studies suggest they do not. E.g. 1/3 accelerated approvals did not meet the benefit-risk threshold for full approvals.

In parallel, we see rising costs of pharmaceuticals.

How do countries deal with this?

#ISPOREurope

In Germany, a drug is reimbursed from the 1st day on the market. The discussion is about the price.

HTA agencies look at the level of evidence alongside of the price, but noting that countries wish to ensure that patients can access new drugs

#ISPOREurope

These new methods are promising but there is a question about how much HTA agencies understand these methods.

Reimbursement decisions should be understandable and predictable. These methods need case-by-case considerations, whereas RCTs have standard methods

#ISPOREurope

Even with these methods, do we still need further studies? Yes, we need studies to confirm. This can be a problem, because rarely studies with the adequate standard follow the marketing authorisation.

#ISPOREurope

In conclusion, we should not waste the opportunity to demand appropriate studies at the time of approval.

Re-evaluation can provide incentives for further studies, which could lead to a price increase if evidence supports it.

#ISPOREurope

Q&A next!
Q: How do you think that uncertainty should be used in decisions, and how can it be linked to post-launch research?

A: Steve has concerns about the overemphasis on CEACs but not necessarily on the consequences of uncertainty to population health.

#ISPOREurope

We should translate the error probability into the consequences on health. We can provide these metrics, to inform whether to ask for further research or to inform price negotiations.

Uwe agrees that there is no place for stats sig in reimbursement decisions

#ISPOREurope

Sylwia notes that, if we have a lot of uncertainty from the prediction from surrogates, if we require re-evaluation for MA, this kind at re-evaluation should also be considered for HTA

#ISPOREurope

Q: How to judge the appropriateness of methods?

A: Uwe answers that researchers should choose & justify the methods, as what happens now. It is nearly impossible to give a recipe of what to do in which case. Also compare the results of different methods

#ISPOREurope

A: Eva notes that in Germany they are discussing CAR T-cell therapy, for which there is few data from few pts, and the therapy has a high price. There is also a discussion over the time frame of the price. Discussion ongoing in Germany.

#ISPOREurope

Q: Is RWE ever sufficient to validate a surrogate endpoint?

A: Sylwia notes that there's not much research done in this area. Her intuition depends on how the data are used and the extent of the risk of bias, and how to minimise its impact on predictions

#ISPOREurope

A: Steve notes the increasing disconnect between regulators and HTA. Regulators have been approving the drugs based on surrogate endpoints, while HTA tend to use PFS or OS, which can be confounded. Need to connect more the regulatory world.

#ISPOREurope

A: Eva notes that there are many endpoints collected in RCT that may be useful in combination with the surrogate outcomes. So surrogates may not need to be formally validated given the other endpoints in the trial

#ISPOREurope

Q: Should g-methods be used over and above propensity score matching always? For HTA purposes, does Uwe recommends asking for various methods?

A: PSM and regression works well for some situations, while g-methods work also for time-varying confounding.

#ISPOREurope

The different methods make different assumptions. If more than one method is valid, then try them both, which is a structural sensitivity analysis, and within the method, should different ways of implementing them, in sensitivity analysis

#ISPOREurope

Q: Often RCTs do not reflect the treatment sequences that are available in all jurisdictions, which makes the standard ITT analysis not applicable here. Do we need to think about adjusting approaches for standard RCTs?

#ISPOREurope

A: Steve notes that this is not a new issue for a reimbursement point of view, but the regulator is now recognising that the trial should reflect how the drug will actually be used.

#ISPOREurope

A: Uwe agrees that ITT analysis is of questionable relevant in many situations, which will hopefully have consequences to how this will be approached by regulators and #HTA bodies

#ISPOREurope

We're now out of time. Excellent plenary session, thank you to all speakers 👏 #ISPOREurope