Thread by @lymanstoneky

What is the best empirical evidence that therapy actually improves mental health?

I am struggling to find anything credible.

This meta-analysis of 147 studies seems to suggest p-hacking is very common, publication bias is huge, and even with that the typical effect of therapy vs. care-as-usual is clinically insignificant. pubmed.ncbi.nlm.nih.gov/21770842/

This more recent one specifically on CBT for adult depression suggests that CBT has been wildly overrated by creative research practices. journals.sagepub.com/doi/abs/10.117…

Seems like even the most enthusiastic meta-analysis yield NNTs of like 3-4. Many studies don't even tell you the actual effect size in a clear way, they just say patients "improved" and it's like okay *but how much*?

Furthermore, it bothers me that we compare therapy to care-as-usual or waitlist as the default condition, or to other-kinds-of-therapy as the alternative, when we should be testing against like "going on a vacation" or "regular diet and exercise" or "taking a sleeping pill"

Somebody needs to do like a 10-arm study with 2,000 people in each where the arms are stuff like "hire a home cleaning service" "send them to a resort for a week" "CBT" "give a fitbit and text message reminders to exercise" "prescribe antidepressants" "prescribe sleeping pills"

And it shouldn't just be depressed people each arm should have like 1,000 diagnosed depressed people and also just like 1,000 random adults.

I assume it costs what like $5,000 per person in treatment to do something like this so it'd be like a $100 mil experiment at the same time some envelope math suggests antidepressant sales in the US are around $15-$30 billion/yr so the money is there.

Lots of meta-analyses seem to report Cohen's d which is fun and all but in the underlying studies it seems often one SD turns out to be a rather small absolute movement on an actual clinical scale, so even a "big" Cohen's d doesn't yield lifechanging results.

I mean at the prices charged I think we should expect evidence.

https://twitter.com/techbasset/status/1479185302929305600

Yeah this is an interested question I haven't seen explored: is therapy more effective for epople who report e.g. few mentors, peers, or family members with whom they discuss personal problems?

https://twitter.com/disluckyperson/status/1479186290679631886

Nice review of studies of exercise: it has similarly small effects. cochranelibrary.com/cdsr/doi/10.10…

It seems like the best read of the evidence on therapy and antidepressants is that the expected effect of any specific treatment is very low but heterogenous, but motivated patients cycle through numerous treatment options until they hit one that works for them.

i.e. you should have low expectations that a specific treatment will help you, but having your case intensively managed by someone who is trying lots of stuff and observing outcomes often leads to improvements

The problem with this is that it creates a massive inference problem. Is it that "eventually you find the treatment that works" or is it "eventually *something* will work because the brain desires to heal."

When you're dealing with sequential treatment regimes selecting on failure of prior treatments and persisting until improvement, it's a really complicated problem to model in terms of causality, and also it raises the question of if "active case management" is the actual cure.

Like, can you cure mental health issues by just convincing the patient that lots of things are being tried, something is being done, etc. Placebo effects we know are large (and growing over time!!) so this seems plausible.

Anyways, gonna stop tweeting about this because lots of people who've totally for sure been cured by therapy of various things are very calm in my mentions now.

See, this kind of response bothers me a lot, because I also know people who were clearly screwed up by their experience of therapy, including some where they clearly think they were helped by it, and they very very clearly were not.

https://twitter.com/BluishCheckMark/status/1479213737060057090

Getting a lot of this kind of reply and not sure what to make of it.

Seems like a way of saying "Mental health literally isn't a valid concept at all."

Which, maybe? But that seems pretty Overton-Window-edy.

https://twitter.com/DavidAOliverJr/status/1479222985039007744

Oh here's a fun paper: clinicians and patients have shockingly little agreement on what symptoms patients are experiencing and how bad they are! sciencedirect.com/science/articl…

Kinda points to one of the challenges here that unlike, say, blood pressure, where it's not *that* hard to come to agreement between parties on what the condition is, with many mental health conditions even defining the condition to be treated is very hard!

responses so far are a mixture of "here consult this guru-therapist who can explain to you why therapy works" or "here's a theoretical research note that doesn't actually contain empirical evidence" or "here's a study where the outcome is significant but the size small"

That last one seems to be a huge issue. I've now read several meta-analyses and a number of specific studies that inarguably find *significant and positive* effects of therapy on average, but when you drill down to the actual effect *size* it's almost trivial.

Like, oh yeah, the therapy condition did 3x better than the control group.... which means on a 55-point scale they improved an average of 0.3 points instead of 0.1. etc etc etc

Finding large, well-powered studies that demonstrate large average effects or even large effects for meaningful subgroups seems to be extremely hard, not least because it seems most of the studies here just *are not extremely large*? Why are they so small?

Is it because the typical practice size involved in a study is modestly sized? A perusal of the studies that looked strongest revealed a lot of studies conducted by one clinic at a time, so sample size is capacity constrained. But that's obviously a problematic approach...

... because "this one clinic" might have a positive effect because of their treatment strategy... or they have a nice physical environment and receptionist! Seems like you need multi-site studies?

A correlated problem here is that small sample size means I've....

... come up dry on some very basic and obvious questions like, "To the extent the effect of therapy varies by individual, what individual traits predict that variation?" Does a person's class, personal background, personality, IQ, sex, or anything else predict if therapy "works"?

To answer this kind of question you need a pretty large sample size and it seems like this kind of subgroup analysis is rare because 1) clinical patients are self-selected along a lot of interesting traits and 2) sample sizes are too small for significant subgroup analyses

But that being the case it makes it almost impossible to know if a person who doesn't want to go to therapy would be helped by being persuaded to go, since all the studies are explicitly of people who *want* therapy with the control group being people who want-it-but-don't-get-it

The other genre of response I've gotten a lot of is people being like "stop coping and just go to therapy." Which is a nice example of the cultish thinking around this issue: you can't question the consensus without revealing that you're unwell!