Thread by @alexandrosM

In this one clip from his latest video, you can see the complete blindness of Vinay Prasad to the cult of RCTs.

He contradicts himself in a messy jumble of confusion because he's making a good point but can't go far enough.

Let me explain 🧵

First, in his rush to defend RCTs, he presents a study comparing meta-analyses of RCTs with meta-analyses of observational trials (or big trials of both types).

He forgets that meta-analyses themselves are observational.
And that includes meta-analyses of meta-analyses too.

So he uses an observational study of observational studies on observational or randomized studies to prove that... what? That observational studies should not be trusted?

You'll need to use another tool for that job, son.
pubmed.ncbi.nlm.nih.gov/15929750/

Then he looks at the results of that meta-meta-analysis of sorts, and sees that observational and randomized trials agree in 2/3rds of the cases. Weird! You'd think that with so much noise, that wouldn't be the case.

But even so, that tells us nothing about which is better.

Then he has his triumphant moment when he finds that the comparisons agree with "statistical significance" in only 1/6 cases.

He forgets to inform us that statistical significance is a completely arbitrary threshold in this case. Because meta-analyses are observational.

And even if it wasn't, it STILL doesn't tell you anything about whether observational trials or RCTs are better.

What you need for that is to see whether RCTs get the right signal AHEAD OF TIME.

This is where the second facepalm moment in the video comes in.

Unfortunately I can't find the AHRQ report he talks about, and he doesn't link it, but from context I surmise that what he is seeing is that an older meta-analysis of RCTs on lung cancer and selenium, and a more recent meta-analysis of more recent RCTs on same are in disagreement

First of all, that doesn't mean the newer meta-analysis is more correct, especially if they dont cover the same studies. Remember, Meta-analyses are observational!

But even if it did, that means that the prior meta-analysis of RCTs was wrong. So why should we trust the new one?

He's using this to claim that even the little concordance that was left is going away, but instead what he accomplishes is to demonstrate that meta-analyses of RCTs are just as bad, and that they reverse themselves just as easily.

So while he's preaching the gospel of how nutritional science is garbage, and he's right about that, the problem isn't with observational studies. The problem is that the brand of statistics used for "evidence-based medicine" is junk, and that includes his beloved RCTs.

The reason nutritional science sucks is the same reason as modern medicine. They isolate a specific question in a chaotic system and try to give definitive broadly-applicable answers. Answers that often doesn't exist. So they "hallucinate". Just like LLMs.

https://twitter.com/alexandrosM/status/1582403670012211201?s=19

The truth of the matter is that the "analytic flexibility" he bemoans in observational trials, is alive and well in RCTs. There is a metric ton of wiggle room to nudge results in either direction. Until the day he comes to terms with that fact, he'll continue to be lost.

This stuff isn't too complicated to understand for people with engineering or similar backgrounds. It's just that the majority of people who understand it are in medicine and there's no money in calling your entire profession out. Or motivation. And those who do get discredited.

So ultimately, what gets presented as sophisticated statistics is just a coping mechanism for a privileged club.

I mean, just think about what a meta-analysis is supposed to do. You have many studies (randomized, even) contradicting each other.

So what are you supposed to do? You plug them into a formula, and supposedly the result should be closer to the truth than any of the studies.

But wait a minute. How is this science? If in physics multiple experiments disagreed, we wouldn't average them out.

Why is nobody asking WTF is going on when multiple randomized clinical trials with sufficient statistical power end up with opposite conclusions? Are we supposed to attribute this to chance? Is something else going on? Is perhaps the answer different for different populations?

Who cares! Throw it into DerSimonian-Laird and hope for the best!

And somehow, it gets worse.

Evidence-Based Medicine (as distinct from evidence-based medicine) doesn't have any notion of cost-benefit analysis.

In EBM as practiced, A deadly adverse event below the statistical significance threshold is not real, but a benefit, even cosmetic, if it clears statistical significance, is recognized.

And thats how you get FDA-approved hairloss prevention medication that can ruin your life.

...and this is why, even someone like Vinay Prasad, who tries hard to be disciplined and stick to strict EBM (most of the time) gets lost in the statistical contortions and starts spouting nonsense.

And in the comments, hundreds of practicing doctors applaud him. Could be yours

I'm sorry I don't have a hopeful answer here. I'm as lost as you are. All I know for sure is that the people spouting numbers can't even keep their story straight. That's something, but it's not much.