Thread by @WiringTheBrain

A little rant: I’ve noticed a trend in some science communication, especially in discussions about neuroscience and psychology research, and I don’t like it…

The move (kind of Malcolm Gladwell’ish but from people who should know better) is to take the findings of some particular study and draw sweeping general conclusions from them

This involves: (i) taking the data themselves at face value, (ii) assuming they are robust and would replicate, and (iii) generalising from some particular experimental set-up to draw implications about real-world behavior

This trend persists even when we know about the methodological problems of fields like social psychology and neuroimaging association studies and candidate gene associations with behavior, and on and on…

And even when we know (or should know) that brief experiments on psychology undergrads in highly controlled and frankly often bizarre set-ups in the lab don’t necessarily have any real-world implications

This style of communication is often accompanied by a rhetorical device: rather than saying: “a study found X” and giving some useful description of it (where we can evaluate things like sample size and statistical methodology and researcher degrees of freedom)…

Some commentators instead switch to a mode that sounds like they’re discussing a generalised truth. Something like: “If you take people and put them in a scanner and show them pictures of X you can see area Y light up if they’re conservative but not if they’re liberal”…

Or: “If you give people lists of words to read that are about old people, they’ll walk more slowly down the corridor afterwards”

Phrasing the findings of particular studies in that way is simply dishonest. It suggests to listeners a degree of robustness and replicability and generalizability that usually hasn’t been shown.

They're taking results and presenting them as rules.

A similar phenomenon: saying studies show that you can “predict” something, like some aspect of someone’s behavior (or some life outcome, etc) from some imaging or personality measure…

Most listeners will (quite reasonably) take the forceful, unqualified use of the word “predict” in a sentence like to mean: “predict accurately in individuals”.

When the real meaning might be: “predict statistically with slightly greater than chance accuracy based on a weak correlation across individuals observed in this one experiment with n=20…”

I recognise that we can’t bog down everything with caveats and qualifiers and hope to retain the attention of the general public. But we also shouldn’t present very particular (often untrustworthy) findings as established facts for the sake of a neat, convincing story.