Thread by @JohnSakaluk

Dear Clinical/Replicability/#clinicalOSlist friends, stay awhile and listen about our forthcoming review of the evidential value for lit. of APA Div.12 Empirically Supported Treatments (ESTs), with colleagues @williamspsych, @kilshaw_r, and Rhyner psyarxiv.com/pzbhw/ 1/16

ESTs are considered gold-standard therapies that have passed certain bar of scientific evidence. Div. 12 further classifies them as "Strong" or "Modest" (sometimes accompanied by "Controversial") to indicate the strength of the accompanying evidence. 2/16

The original criteria for determining EST-status were published by Chambless et al. (1998). This was a watershed (and positive!) moment for clinical trial psychotherapy research. Strong designs and evidence-based practices are important! 3/16

Still, the replicability crisis has shown us that 2x p < .05 is a precarious standard of evidence, esp. when the rules of NHST are ignored. So we wanted to know: how strong is the evidence underlying these ESTs? And what does it mean to have "strong" vs. "modest" evidence? 4/16

Conveniently, Div. 12 maintains a website of all current ESTs, their evidence level designation, and the clinical trials/"key references" used to determine their status (div12.org/treatments/). We used this as our predetermined sample of articles. 5/16

We looked at four evidential value indexes: rates of misreporting (i.e., statistical typos), statistical power, estimates of replicability, and Bayes Factors. None, individually, is a "truth serum", but combined they paint an interesting picture about research credibility. 6/16

Tl;dr: we thought ESTs (and ones classified as "strong", in particular) should be accurately reported and well-powered, shouldn't have evidence of inflated rates of significance, and should have data that are more compatible with efficacy than with a null effect. 7/16

All in all, we coded 3463 effects from 453 articles, though many of these were not reported in suffiicent detail to be useable to calculate some/all of our indexes. Raises an important question of how, then, the same effects could be deemed reliable evidence of EST efficacy? 8/16

Analytic flexibility is another problem in the EST literature. Despite similar designs, articles often varied re: what effects were deemed "key" by authors. Main effect of condition? Time? Condition x Time interaction? Seemed to depend largely on what was p < .05. 9/16

As for our four indexes, see breakdown across the sample (Table 1), across individual ESTs (Table 2), and across evidence labels given by Div. 12 (Table 3). Lots going on (especially for a short report), but here are a few highlights: 10/16

Do any ESTs perform consistently well? Yes! Roughly 20% of the ESTs we reviewed faired well across most of our metrics. These included 22% of "Strong" ESTs (e.g., Exposure Tx for Specific Phobias), and 14% of "Modest" ESTs (e.g., Reminiscence/Life Review Tx for Depression). 11/16

How many ESTs performed consistently poorly?: 56%. These included 52% of "Strong" and 64% of "Modest" ESTs, and a number of therapies that occupy go-to-treatment status within the clinician community. 12/16

One of our big-picture take-aways: methods reform is needed in EST research. Verification of accurate reporting, Reg. Reports, and large-scale collaborative/big-n studies needed if "ES" in "EST" is to mean what we want it to mean. 13/16

Also: Might be time to consider "psychological reversals" (h/t @VPrasadMDMPH & @adamcifu). If widely-practiced tx for given disorder lacks evidence of benefit and/or cost-effectiveness, may be time to invest more in tx that is better evidenced/more cost-effective. 14/16

Lastly, strong evidence is not just needed for claims of tx efficacy, but also for claim of/against tx harm. As a teaser, we're nearing submission on a similar evidential value synthesis looking at tx's identified in Lilienfeld's (2007) PHT paper. journals.sagepub.com/doi/full/10.11… 15/16

Final big thank-yous: @JnfrLTackett and @JMill_er for editing the Special Issue, our reviewers, @MicheleNuijten, @katiecorker, @EikoFried, and @JoshuaGrubbsPhD for their consulting, @BrianNosek for his wisdom, and @dirckk for suggestions for the MS /fin

Sorry folks. @PsyArXiv mods think we posted APA's official version and have rejected the preprint (your template is v. convincing @bmwiernik). Will report back when it's live...