Thread by @KunstJonas

Another unblinded CBT trial from Chalder, who is known for her methodologically-disputed work on CFS, finding no effect on primary outcomes among people with "medically unexplained symptoms." Some effects are observed on secondary outcomes, but... 1/

https://twitter.com/davidtuller1/status/1406335507953422336

correction for multiple comparisons was not applied. The authors state: "Throughout this paper, we present unadjusted p-values. Methods for adjusting the family-wise error by methods such as the Bonferroni correction are known to be conservative..." 2/

"however, if one were to use a method that controlled the false-discovery rate such as the Benjamini–Hochberg procedure then the differences on PHQ-15, WSAS at 20 weeks and CGI remained statistically significant and would therefore be considered as discoveries after..." 3/

"...correction for all nine outcomes (eight secondary plus primary outcome)." So after their correction, the authors find that only 3 out of 9 tests are significant. This is already quite sobering. Yet,.... 4/

the BH correction is probably the least stringent they could have picked. When I control for 9 tests, using a range of common correction methods that are not as conservative as Bonferroni, I get no significant result at all. /5

(Disclaimer, this was only a quick and dirty re-analysis, so I may have missed some things. However, I think the tendency is clear. Effects are not robust to p-value correction.) /6

In their abstract, the authors conclude "We have preliminary evidence that TDT-CBT + SMC may be helpful for people with a range of PPS." But as any reader can see, this should be far from the take home message from this research. /7

Rather, the findings seem not to be robust and suggest that this type of intervention seems to have little or uncertain effect in this type of population ("medically unexplained diseases"). Unfortunately, this spin has detrimental consequences for research and practice. /8

It keeps researchers conducting studies that are doomed to fail. This costs society a lot of resources, leads to little scientific advances, and prevents research shift into more promising directions. However, most critically, it produces work that can misinform treatment. /9

I believe that CBT is great for many things. However, we need to acknowledge its limitations and the bias inherent in research (non-blinding) that becomes critical especially when effects are relatively weak as they often are in psychological treatment research. /10