Thread by @GorwayGlobal

A Thread.
My column in @newslaundry criticised the RCT of Coronil done by Patanjali on, among other grounds, its small sample size. Many have asked me to say what sample size I would have recommended. So here is a #Tweetorial in RCTs and sample sizes. newslaundry.com/2021/02/22/pat…

Imagine a conversation between an Ayurvedic Researcher (R) and a Statistician(S) . It might go something like this:

R: "hey I need a sample size calculation. In my last trial I had 100 patients and the critics laughed, ‘it’s too small’, they said. "

S: Okay lemme help.

S: What is the outcome measure you seek to study?

R: Virus clearance at Day 3

S: I see. So, what is your best guess of the virus clearance rate at Day 3 in the absence of any treatment?

R: I dont know. We think its is 50%

S: That sounds high, I hope you are counting the days properly What is Day 0 for you?

R: Day 0 is when my subject comes to me with a positive RT-PCR.

S: tha's when YOU 1st have evidence of virus +ve status. It’s not the first day of infection. Have you looked at Lead-time bias?

S: While you think abt that, Lets go with the 50% estimate. Now, What is the smallest change on 50% that you think is clinically important NOT to miss?
R: From 50% background rate of clearance at day 3 we think a 70% clearance would be a useful improvement.

S: So lets agree a difference of 20% either way from the background 50% would be the minimum difference that you would not want to miss detecting..
R: (looking confused ).. I dont get this 50% +/- 20% business. Its an Ayurvedic drug how can it be worse than a placebo?

S: Look, park that for a bit. Can we agree that you’ll follow the usual 5% alpha error rate and 20% beta error rate.

R: What do they mean?

S: Okay, listen carefully this can be a difficult tghing to grasp at first.

S: Well the 5% alpha error means that in the experiment you’re about to do, assuming your coronil is no better or worse than a placebo, you are willing to run a 5% risk that you (erroneously) conclude that Coronil is different than the placebo . Hence p<0.05, get it?

R: And the 20% beta error rate?

S: That means that if Coronil is in reality not the same as placebo then you're happy with a 20% chance that the experiment you are about to embark on will fail to show a difference and lead U to conclude (erroneously) that Coronil is worthless.

R: But hang on, Coronil is an Ayurvedic drug. It cannot be worthless. Ayurveda has been around for millennia. The Vedas have described it.

S: Every researcher says something like that. " My drug HAS to work", they say. But you're a scientist now about to do an experiment.

S: You have to think like a scientist, keep an open mind. Be sceptical. Be objective. But if you are very sure that coronil is as nectar from the Gods, why do the experiment at all? You have to be open to all possibilities when you do an experiment. Agnostic about the result.

R: This is all so confusing. I just want to prove that my Coronil works, tell me a number for the sample size.
S: The trial you are going to do does not PROVE anything, it is merely to gather evidence for or against the null hypothesis that Coronil is the same as a placebo.

R: That’s not what my Acharya told me. He wants to publish a paper in Phyto-something to show that coronil does some good. Why don't you give me the sample size number that I need to recruit.

S: Oh well okay then, let me help, Let me recap where we have got to.

S: You think the background rate of virus clearance at day 3 is 50%. You don't want to miss a difference of 20% either way. And we agree that we fix the alpha eror rate at 5% and the Beta at 20%.
R: Right. If you say so.
S: No No YOU gave me those numbers. It's your experiment.

S: I think 20% is too high. Surely a 15% diff is worth detecting. Lets set it at 15%, please. If Coronil really offers a 65% viral clearance at day 3 you want a to design an experiment NOT to miss that big a difference with more than a 20% prob.
R: okay lets go with that.

S: Brilliant. I’ll plug those numbers into Epicalc this free software from EpiInfo from the guys at Emory University School of Public health in Atlanta, Ga.

R: The guys who wrote this software at Emory; do they know anything about Ayurveda?

S: I guess not.

S: but then they dont know much about cancer or modern medicine either, they are statisticians you see, they design statistical experiments and do the analysis. An RCT is a statistical experiment.

R: okay lets go!

S: Ive plugged those numbers and the sample size I get is (pic)

S: You'll need to recruit about 360 patients, 180 in each arm of your trial to be able to demonstrate a +/- 15% (or greater) departure from the background 50% virus clearance that you expect in the absence of any treatment.

R: That’s bonkers, Coronil works, it says so in all the ancient texts I’ve read. I am not going to do a trial in 360 patients give half of them a placebo that might let them die. This statistics is all a western con trick!!!!

S: I am sorry you feel that way. That’s what I said at the beginning. If you’re so sure Coronil works and you cant face the prospect of putting it to the test then you should market it on the basis of faith newslaundry.com/2020/11/11/ayu…

Note: this entire tweet thread is fictional. All researchers react with shock and disbelief when a statistician gives them a sample size number. But just as you need a BIG telescope to detect a far away star, you need a large sample to detect small effects.