Thread by @alexandrosM

I'm realizing that the insitence on Randomized Controlled Trials (RCTs) as the only evidence that matters when deciding if a medicine/supplement should be used, structurally biases against generics, over-the-counter meds/supplements, and those with few side-effects. Here's why:🧵

The first class of problems has to do with wide availability when the subject of effectiveness on a new disease is raised.

1. Cheap OTC generics with few side-effects get used a lot in an emergency, where word of mouth spreads, making it much harder to form a control group.

2. These substances, when there's a suspicion they can be effective in an important disease, will spark many studies all over the world. This means there will be many small trials, of varying protocol/dosage and study quality. This is a big problem for two reasons:

2a. Varying dosage/protocol means that the results will necessarily be suboptimal. Even if some studies use the perfect protocol, others won't

2b. Some studies will be poor quality and/or fake. Critics can focus on these to invalidate every other study by diverting attention.

3. However, that's why we have meta-analyses right? Well, for something this controversial, there will likely be many meta-analyses, allowing motivated actors to pick and choose which one they prefer.

3a. Meta-analyses can focus on only "statistically significant" trials, or only on RCTs, and in general play with inclusion-exclusion criteria. Setting the criteria too tight, allows some analyses to remove most of the data, concluding "no evidence exists to suggest...".

3b. As some trials get shown to be of bad quality, calls for retraction of other meta-analyses get made, delaying conclusions. For a meta-analysis to be updated, it would need resubmission that can take months, even though the update takes a few minutes.

3c. Meta-analyses can water down the results of the strongest trials by averaging them out with the weakest trials. When the appropriate dosing is not obvious, this can significantly dull the effects seen.

4. Of course we shouldn't forget the obvious: RCTs are arduous, expensive, and doing one that's big enough to be "convincing" is prohibitively expensive.
A napkin calculation of what it would take to do a study for a 70% effective early treatment for COVID turned up a $10m cost.

Naturally, a new medication that is not available to anyone before the trial has none of these problems, and since it's going to be patented, budget is not a problem. The sponsor can do small unpublished studies to optimize the protocol, and then do a big one to showcase effect.

And if there's only one study to speak of, none of the meta-analysis problems apply.

This is why Obama's head of the CDC, Thomas Frieden, writes about moving beyond the RCT: nejm.org/doi/full/10.10…

Simon's perspective is always on point. Sadly he's being hidden under a filter, so here he is for your enjoyment.

https://twitter.com/sival84/status/1447297124052443144?t=epfIDmQxUqrtix198Iw2Ew&s=19

Fascinating point about the steep, stepwise shift in cost and complexity. For most things they will either be too weak, or overkill.

https://twitter.com/JohnColeman675/status/1447440941879463941?t=7z0mIcGLPjpxsBLTn6cwaQ&s=19

Meditating on this, I suspect what we need is a meta-analytical framework that doesn't penalize a hypothesis for having been studied *too much*, that takes dose dependence into account. I don't know if it exists, but if it does, I bet the word "bayesian" is somewhere in the name.

I've put together a proposal for how to approach this for exactly that class of medicines and supplements, by using the same technology we use to optimize ad delivery and self-driving cars, but I'm not a medical researcher, so take it with a grain of salt.

https://twitter.com/alexandrosM/status/1435565853466972164?s=20