Thread by @marosenbergCO

Can we talk for a minute about the randomized controlled trial (RCT)?

No one can refute that RCTs are one of the most important innovations in the past 50 years. 1/23

Although the famous RCTs are known for overturning existing dogma (CAST study, intensive glucose control in diabetes, hormone replacement therapy…), less famous RCTs have led to thousands of treatments applied daily (each linked to a specific RCT demonstrating benefit) 2/23

However, while saying ‘we need more RCTs’ sounds great on Twitter and in the *Future Directions* section of papers and grants, the reality is that most clinical and health policy questions are poorly fit for application of the RCT. Here’s a few reasons why… 3/23

Let’s start with enrollment. Enrollment in an RCT is based on equipoise, meaning that any patient who meets the study enrollment criteria should have the same probability of receiving treatment or control. 4/23

While this sounds great in theory; in practice, there is almost always a gap between the broad enrollment criteria of a study and the specific clinical characteristics that an experienced clinician uses to guide decisions. 5/23

As a responsible treating physician, if I think that a patient needs a treatment, I’m going to apply that treatment, not wait around and see if they get randomized to that treatment within the RCT. 6/23

Obviously, for drug trials where the new drug wouldn’t otherwise be an option, this isn’t an issue, but for most procedure studies the procedure is already available, and often clinicians already have thoughts about who might or might not benefit from it. 7/23

Frankly, you can argue that it would be unethical to enroll a patient in an RCT for a treatment if you were certain that they need that treatment (see Parachute RCT). As a result, RCTs of procedures seem to almost invariably have null results due to this selection bias. 8/23

What about the interventions being studied within an RCT?

Again, for a drug trial, sticking to a specific treatment is relatively trivial (although real-world adherence is a legitimate concern). 9/23

There’s no operator experience, general knowledge, or technology improvements that can change over the course of the study to impact how the drug works; it’s the same drug at the start of the RCT as at the end. 10/23

For procedures and policies, this just is not the case.

Procedures evolve over time, especially if they are new enough to warrant an RCT in the first place.

Operators get more experience, technology improves. The procedure can look quite different at trial end. 11/23

Policies are never implemented in a vacuum, and if you’re lucky enough to be able to complete the study before stakeholders issue a mandate that everyone needs to follow the policy, it’s still very difficult to avoid having key policy concepts bleed into the control group. 12/23

For example, if I told you you’re going to be the control arm of an RCT to study an intervention to reduce hospital readmissions, you’re very likely to take measures to reduce readmissions on your own. 13/23

The result, again, is that RCTs focused on procedures and policies can rarely find a significant difference between groups, leaving investigators and clinicians to argue over study characteristics in a manner no different than if the study was observational (see CABANA). 14/23

Finally, consider how we analyze an RCT. For most, the analysis is some form of time-to-event comparison, with or without accounting for ‘known’ risk factors, where everything hinges on whether the p value is above or below 0.05.

15/23

Of course, if the treatment is a drug started at one point in time, with follow-up until a single obvious clinical event or death, this approach is perfectly fine. 16/23

But let’s face it, is this how we treat most conditions? Do we tell patients, "Start this drug, and in 5 years we’ll see if you’re alive or not"?

Of course not.

17/23

We administer the treatment, monitor results in near term follow-up, make an adjustment, monitor results, rinse-and-repeat...

But the methodology of RCTs can rarely capture this level of dynamic change. 18/23

At best, you can try for some sort of factorial design, knowing that with each branch point you’ll need to increase the sample size exponentially. 19/23

We analyze an RCT as though we’re studying two Petri dishes in a lab, in which the only possible difference is the compound added.

In reality, RCTs are conducted in the real world, where it’s not okay to let bad things happen to people just for the sake of science. 20/23

Saying in response to a complex policy or clinical question, ‘we need to do an RCT,’ is no different than responding to a natural disaster with, ‘we need to stop climate change.’

Sure, an RCT for every clinical decision would be ideal, but the real world is too complex. 21/23

Personally, I think that we'd be better off learning new ways to evaluate clinical decisions.

Bayesian statistics, structural models, reinforcement learning... approaches most clinical investigators didn't cover in the stats section of their MPH...

22/23

Open for thoughts and rebuttals...

@drjohnm @ADAlthousePhD @venkmurthy @VPrasadMDMPH @rwyeh @f2harrell @statsepi @RogueRad

23/23