(THREAD)
A few months ago, the Annals of Medicine published a controversial piece titled “Why all randomized controlled trials produce biased results.” Topic: not a bad idea - we should examine trials carefully. Execution: left something to be desired.
A few months ago, the Annals of Medicine published a controversial piece titled “Why all randomized controlled trials produce biased results.” Topic: not a bad idea - we should examine trials carefully. Execution: left something to be desired.
We have penned a reply that covered some of the most problematic misstatements, with helpful input from @coachabebe, @GSCollins, and @f2harrell
After some emails between ourselves, Krauss, and the journal, he chose to revise the original piece in response to some of our comments
However, much content with which we disagreed remains, and the journal agreed to publish our response (revised after Krauss’ revisions) as an editorial. This is now live on the Annals of Medicine website, link here: tandfonline.com/doi/full/10.10…
There are just a few points from the editorial I’d like to highlight here
As well as a few that were cut from the final editorial for space constraints
Points that made it into the editorial which I’d like to highlight:
1) Krauss states that the simple-treatment-at-the-individual-level limitation is not thoroughly discussed limitation of RCT
1a) first, this isn’t a reason to state that all RCTs produce biased results. It has literally nothing to do with “biased results” in RCTs. This was/is a common theme. Many of Krauss’ highlighted points have little or no relevance to “biased results”
1b) this also ignores a large body of work in complex, adaptive, innovative trial designs
1c) consider I-SPY 2: ispytrials.org/i-spy-platform…
1d) or STAMPEDE: stampedetrial.org
1e) or REMAP-CAP: clinicaltrials.gov/ct2/show/NCT02…
1f) there is in fact a great deal of work being done to expand trial designs beyond simple parallel-group treatment comparisons. What do you think @SCTorg exists for?
2) Krauss spends much time discussing the issue of “achieving good balance” in RCTs
2a) this has been discussed at great length previously by Altman, Senn, and others over the years
2b) For example, here: onlinelibrary.wiley.com/doi/abs/10.100…
2c) I also like this quote from a more recent piece: ncbi.nlm.nih.gov/pubmed/29518478
2d) “Under proper random treatment assignment, distributions and imbalances of all baseline covariates among treatment groups are random. Therefore, random baseline covariate imbalances and random treatment assignments must be accepted or rejected, together.”
2e) “Pursuing perfect baseline covariate balancing under the name of randomized controlled trial is self-deception. Random baseline covariate imbalances can and should be taken care of by statistical approaches. Perfect baseline covariate balance is not necessary.”
3) Continuing with the balance issue: even after the corrigendum, the piece still contains this troubled statement in the conclusion
3a) “No researcher should thus just generate a single randomisation schedule and then use it to run an experiment.”
3b) “Instead researchers need to run a set of randomisation iterations before conducting a trial and select the one with the most balanced distribution of background influencers between trial groups”
3c) Putting aside whether balance is necessary, this is not remotely feasible in the majority of RCTs in medicine. It is maddening that this statement went past peer review unchallenged.
3d) Plus, if one wants to apply a balance-promoting method, there are better options than “recruit all the patients, randomize and check for balance, then just randomize again if the first one isn’t perfectly balanced”
3e) that’s not even truly “randomization” any more - if you’re discarding the first randomization schedule because you don’t like it, then we are no longer truly “randomized” since one option was discarded
3f) if one does want to assure a degree of balance on covariates – which may be desirable in some instances - there is an entire world out there on covariate-adaptive randomization procedures, which actually CAN be carried out in the real world
3g) Here’s one of a score of articles on the subject, with tons of references to other papers in this area: sciencedirect.com/science/articl…
Some very smart people have been hard at work on this:
It’s a shame that this entire literature from statisticians and trialists was passed over as people uncritically shared this piece.
Points that were cut from editorial, but should be noted
4) Krauss also mentions the placebo-only or conventional-treatment-only limitation
4a) this does have some merit: trials should include “control groups” reflective of the current standard of care
4b) but it’s a little silly in the specific context of the trials he criticized – some of the trials he criticizes for comparing against a placebo were compared against placebo because there was no other available treatment at the time
5) Some of the other statements are just baffling or misleading
5a) for one example, this passage: “The authors of this trial state that the results could apply to about 3% of the US population.”
5b) I went and looked at the reference linked to this statement.
5c) The trial he references with this statement is the Diabetes Prevention Program: ncbi.nlm.nih.gov/pubmed/11832527
5d) The authors made no such statement about “3% of the U.S. population”
5e) The closest: “An estimated 10 million persons in the United States resemble the participants in the Diabetes Prevention Program in terms of age, body-mass index, and glucose concentrations, according to data from the third National Health and Nutrition Examination Survey.”
5f) Which means the author got out his calculator and divided 10 million by the U.S. population so he could state that the findings only applied to 3% of the U.S. population
5g) But that isn’t what the authors said, nor is that statement likely even true, and even if it IS true, it still doesn’t make a whole lot of sense.
5h) the Diabetes Prevention Program was targeted at…you guessed it…PREVENTING DIABETES
5i) it makes sense to focus such an intervention on the population at highest risk for diabetes (in this case, overweight adults with elevated fasting glucose)
5j) the authors made this statement to (correctly) emphasize the LARGE number of potential beneficiaries of a DPP-like intervention were it scaled up to the entire population
5k) Krauss deceptively turned this on its head to make it sound as though this trial was not useful because its findings don’t extend to many people
5l) RCTs are always by nature going to be focused on the treatment of a specific population. In many cases, this is a feature, not a bug. We want to study how interventions work in people at highest risk of a disease or event occurring
In summary, as we stated in the conclusion of our editorial
We are not intending to suggest that RCTs are unimpeachable. Quite the opposite: RCTs must be planned with careful consideration of the requisite assumptions, monitored with extreme rigor, and analyzed properly to ensure valid statistical inference from the results.
We simply prefer that discussion of RCTs be adequately informed and concentrate on legitimate questions and solutions. We hope that this leads to better-informed discussion of true strengths and weaknesses of RCT’s in practice.
