Thread by @psybalazs

Does blinding work in #antidepressant #RCT? Recently there were 2 papers on this topic with opposite conclusions. A🧵on #blinding and why #robustness #DataAnalytics may lead to better #science

The two papers are tinyurl.com/2p9atk4t (p1; by @AmeliaJScott, @LSharpeUSYD and @BenColagiuri) and tinyurl.com/mtmtx7m3 (p2; by @Toshi_FRKW and colleagues).

Their two conclusions:
-p1: "meta-analysis suggested that blinding was unsuccessful among participants and investigators."
-p2: "patients or assessors were unlikely to judge which treatment the patients were on."

What explains the difference? Its immediately apparent that they analyzed different trials: p1 identified 7 trials, while p2 identified 9 and there are only 2 (!!!) trials that are included in both analysis. This difference is due to differences in inclusion/exclusion criteria.

Few other differences:
- p1 uses Bang's blinding index, while p2 uses kappa statistics
- p1 excludes double dummy trials, p2 includes them
- p2 analyzes trails 2000-2020 only, p1 has no time restriction
- p1 includes trials with substance use comorbidity, p2 excludes them

I have strong feelings about some of these analysis decision, but not for all. Lets just agree that for all decision points listed above either option can be reasonably justified. The important bit is that these seemingly small differences lead to polar opposite conclusions.

What's the way forward? IMO either we argue over each of these analysis decisions until we form a consensus and come up with the one and only True analysis OR

we embrace the multiplicity and focus on the robustness of results: instead of presenting results of a single analysis with a single set of choices, we should explore all 'choice configurations', i.e. the 'space of reasonable analysis' (eg: tinyurl.com/52ftp9kk)

For example, above I gave you 4 analysis decision points, leading to 2^4=16 reasonable analysis (note, there are other factors as well, thus, realistically there is a much larger number of reasonable analysis options).

Why not examine all 16 (or even more models) and assess in what % of the cases blinding worked? Such robustness analysis reduces the influence of each individual analysis decisions and encourages a big picture view of the problem.

Right now we have a divergent conclusions from these studies. Maybe the conclusions would converge if they both have explored their respective analysis decision space. Maybe not, but I think it would be worth exploring.

So, are AD trials blinded? My take is that the correct guess rate in AD trials is almost always <60% (50% is a perfectly blinded trial). So even if blinding is broken according to some hypothesis test, the effect is likely to be small.

For comparison, the correct guess rate in #psychedelic #microdosing is ~70% and probably >90% in macrodose trials. Thus, the lack of functional blinding potentially has larger influence on #psychedelic trials compared to conventional #antidepressants RCTs.

ps.: I do like both of these new papers and this🧵is not a criticism, rather it aims to be a constructive thinking out loud of why results are divergent and what can we learn from them

ps2: this may be of interest to you @_fernando_rosas @MattBurkeMD @DavidErritzoe @ProfDavidNutt @a_lisinski @LironRozenkrant @eturnermd1 @IshratHusain_ @JeremyHowick

(and ☝️ might be of interest to @sdpnayak as well)