Thread by @mcmullarkey

If you ever want to sound like an expert without paying attention, you only need two words in response to any question

"It depends"

A thread on why we should retire that two word answer 🧵

When people say "it depends" they often mean the effect of one variable depends on the level of at least one other variable

For example:
You: Does this program improve depression?
Me, Fancy Expert: Well, it depends, probably on how depressed people were before the program

Understandably you'll want some evidence for my "it depends"

Luckily my underpaid RA has already fired up an ANOVA or regression, and *I* found that how depressed folks were before the program moderated the effect of the program

"It depends" wins again?

Nope, so many problems

1. Correlation still isn't automatically causation

Unless you've experimentally manipulated the moderator, you have a lot of work to do before you can claim that a moderator is causal

I recently did a thread on some bare minimums, and they're TOUGH

https://twitter.com/mcmullarkey/status/1443666896763633667?s=20

Ok, but now using my expert-ness I've smartly retreated to claiming my "it depends" isn't causal. It's predictive, descriptive, or whatever our audience will find most convincing. We're good now, right?

Not even close

2. "It depends" is too vague to be useful

You realize I, fancy expert, am on the ropes. You ask "Is your 'it depends' about slopes or correlations?"

I sputter until you show everyone the "Same Slope Does Not Imply Same Correlation" section of this paper
journals.sagepub.com/doi/full/10.11…

The technical details in that (excellent and thread motivating!) paper by @dingding_peng and @rubenarslan are super interesting, and I recommend you read

But even if you don't, the bottom line is: Even if "it depends" is technically true we have miles to go before it's useful

But I didn't get tenure for nothing, I've now formulated my "it depends" into a tight, formalized theory. There's equations and everything!

To defend my fancy expert self, I've now put in way more work than most folks do. And it's still not enough

3. Our samples likely aren't large enough to reliably detect moderations with realistic effect sizes

I've beat this drum many times before, so I'll be short. We're often underpowered for main effects and for interactions we need even better power
statmodeling.stat.columbia.edu/2018/03/15/nee…

This bears out in large scale analyses, where interaction effects with personality variables were less replicable than main effects

(Shout out to @EmorieBeck for this super impressive mega-analysis!)

proquest.com/openview/82e1c…

But my expert-ness protects me again! I've gotten BIG GRANT EVERYONE WANTS and hired a data scientist to make sure I'm properly powered. Will you leave me alone now?

Lol no

4. We use the wrong tools

We're measuring everything on Likert (1-5, 1-7, etc.) scales. But since our stats training makes no sense we've been using linear regressions for continuous data the whole time

We need an ordinal regression instead, and when we use it this happens

Screenshots from this paper again!

journals.sagepub.com/doi/full/10.11…

Ok, ok, but now it's like 6 years later and I've run an appropriate model in an enormous sample. The interaction is still there! Sure, the moderation might or might not be causal, but vindication for the expert!?

I mean, maybe

5. We care about practical effects, not statistical ones

The original "it depends" has implied gravity behind it. We might even select who gets the program based on how depressed they are!

Except using single variables to do that is a fool's errand
ajp.psychiatryonline.org/doi/full/10.11…

There's also at least some evidence that single moderator variables have smaller effect sizes than main effects

psyarxiv.com/c65wm/

If we want to make any practical progress in identifying "what works for whom?" or any moderator-centered question, I (real me) think we have to:
- Embrace humility
- Run appropriate models that can account for many higher-order interactions at once

doi.org/10.1080/153744…

Ok, my fancy expert alter-ego is nearing the end of my illustrious career now. I've folded my initial one variable moderation idea into a machine learning model with lots of variables now!

Science loved it, so we must finally be done?

I wish

6. Even once we have multivariate prediction models, we have to directly test whether using those models actually improves outcomes we care about

For example, do people assigned to programs via our model actually get better more than folks randomly assigned to those programs?

"It depends" is almost always too vague, poorly tested, and lacking in practical relevance to be a useful statement, even if it's technically true

What should we say instead?

I don't have a great answer! Some variations I've tried:

"I'm not sure, what kinds of variables might be relevant?"

"Hmm, how could we design a study/run a model that helps us get closer to knowing who benefits more?"

"I don't know"

Also, let's be real, "it depends" could be ok if it weren't so often said with a tut or a smirk

Knowing the world is complicated is no substitute for using the methods necessary to grapple with that complexity

So the next time we're tempted to just say "it depends" to look like an expert, let's all work together to go beyond that surface level response

I bet y'all will come up with some great alternative responses, and I look forward to learning what they are!