Thread by @MikeBradburn2

On international #clinicaltrialsday, it's worth* trial advocates (of which I'm one) taking a step back and acknowledging some of the limitations of the RCT.

* I think it's worth it and I've got a pre-print to publicise.

Here goes...(1/20)

Our paper asked: do RCT participants match their intended target population, and does it matter if they don't?

Short answers - probably not, and MAYBE not. There's many arguments and counter-arguments on this. I won't get them all in. (2/20)

jclinepi.com/article/S0895-…

Firstly, there's plenty of literature saying RCT participants are different to non-participants. Below is a convenience sample of *reviews* which show this. Some of these findings may spurious (multiple comparisons etc), but... at face value, it's a bit concerning. (3/20)

But importantly, this doesn't itself invalidate a trial! Having a non-random RCT sample doesn't mean the effect estimate is wrong.

To illustrate: suppose a trial has too few men and (rather like college aged Mike) a notable lack of women. (4/20)

Now suppose men and women have the same treatment response i.e. E(difference|male) = E(difference|female). The proportion of males:females doesn't impact on the findings here.

But, we're not off the hook yet (5/20)

The homogeneity assumption is questionable. Some argue it's so unlikely, non-random samples can't be extrapolated unless the team demonstrates homogeneity. Note also subgroup analyses have severe limitations for typically sized trials (6/20)

mja.com.au/system/files/i…

Other argue effects are often "homogeneous enough" that the sceptical reader (rather than the investigators) should be tasked with proving otherwise.

There's lots of references around these counter-points, but this blog is a good start : (7/20)

fharrell.com/tags/generaliz…

For me, the single most important viewpoint is of those who decide whether to act on the evidence: the clinicians. I'll take a guess that many will ignore trials whose settings don't match their own - and won't want methodological debates.

Can we do anything about this?
(8/20)

We can maybe learn from surveys, which often quota sample (representation by design) and/or up-weighting the under represented in the analysis. The first is hard; the second, less so. See, e.g.:
(9/20)
rss.onlinelibrary.wiley.com/doi/abs/10.111…
academic.oup.com/aje/article/18…
ncbi.nlm.nih.gov/pmc/articles/P…

So we dipped out toe in the water.

The REPOSE RCT was embedded within the UK-wide DAFNE educational programme and we had access to a database of everyone who went on this course. This gave us information on who came into REPOSE and who didn't.

(10/20) bmj.com/content/356/bm…

The database was NIHR funded and intended for research purposes - it contains more details than a registry typically would. I'll come back to this.

We wanted to use external data to assess 1) was the RCT random; and 2) did it matter to the findings of REPOSE?

(11/20)

1) No and 2) barely. We can be more confident in the trial findings.

That said, don't under-estimate
i) the hassle of identifying and getting external data (DAFNE was good on this - not all data providers are)
ii) the data quality (see i)
(12/20)

iii) getting a "good enough" model to re-weight with
iv) the potential instability of re-weighting models (particularly in the tails of the overlap)
v) The spectre of getting a different answer to your expensive RCT.

It's a lot easier not to look.
(13/20))

Remember the point about oversampling based on sex? Homogeneity in relation to sex is plausible and testable.

Now suppose the people we under-represent are "less engaged", "less happy" or such qualities. Now, not so simple (analytically - or philosophically) to "adjust" (14/20)

Our reference data allowed us to look at this - a bit. As well as the usual demographics and glycaemic control, we also had quality of life, treatment satisfaction and other markers of - for want of better words - "the person" rather than "the disease". (15/20)

In our trial, pre-trial treatment satisfaction was (marginally) higher in the RCT participants: they were more ok with current treatment yet volunteered anyway. There were differences in physical & mental QoL too. Though again, nothing to overturn the treatment effect (16/20)

But this data isn't usually available, so if you go down this route, other proxies are needed. Here at ScHARR, @AbuAlshreef is looking into similar-ish methods which extrapolate RCTs onto less adherent populations (17/20)

sheffield.ac.uk/scharr/section…

Oh, and we considered generalisability only in terms of the people who entered and didn't enter our trial. Other related issues exist, like: is the RCT intervention the same as it would be elsewhere? We haven't addressed these. (18/20)

I'm obviously of the opinion that RCTs are a good idea: for all their biases and limitations they're the least biased and limited option for getting a good answer - when designed well. But we should still challenge ourselves to think how they could be made better (19/20)

If you've made it this far, I admire your stamina and you might want to follow some other (more knowledgable!) people who know a thing or two about this: @stats_tipton @Lizstuartdc @f2harrell

(Fin)

(also on international clinical trials day: have good data management that call out your missing or spurious entries. Should say "too MANY males", thanks @timchater back to the thread)