Thread by @Moshe_Hoffman

Best criticism of social-psychology I have seen 👇.

(By @PsychRabble et al)

Allow me to summarize the key points.

And two important takeaways.

(Thread)

dropbox.com/s/elimpj5u0aae…

The key point:

Fundamental problem in social psych (in addition to p-hacking):

Researchers, motivated by the “wow factor,” read into their (usually cool & interesting) results, the most shocking interpretation that’s *plausibly* consistent w/ their data.

Problem is:

Plausible ≠ justified.

Plausible leaves a lot of room for motivated reasoning.

Which few in field have an incentive to ever call out. Or raise more sensible interpretations.

Cause they also wanna teach and preach the “wow.”

Jussim et al. provide many examples, and discuss key “tricks”:

Eg:
-Show situations matters.
-Conclude people fundamentally misattribute cause of variation to personality.
-W/o showing personality *doesn’t* matter. Or matters *less* than situation.

Eg:

-Show correlation btwn x and y.
-Imply x is *primary cause of* y.
-Hide fact correlation minuscule.

Jussim et al argue many problems come from:

-loose definitions of constructs (what exactly is the fundamental attribution error purporting? Can you state in way that’s falsifiable?)

-reporting standards that allow for misleading statements (eg don’t show summary stats in table)

Another example:

In steretype threat lit, researchers claim no race gap in testing when race not made salient but obscuring fact “race gap” is based on residuals from regression that controls for past test performance. <—very misleading

See below figure, Jussim et al pull from classic stereotype threat paper.

(“non diagnostic” bars refer to no threat control)

Notice: data appears to show no race gap in no threat control treatment.

But graph based on residuals after controlling for past test performance!

Of course there isn’t a gap when controlling for prior test! Once control for prior test no longer really talking about race gap in performance. Talking about race gap in *change* in performance between tests.

Notice: misleading. But not misleading in arbitrary way.

As Jussim et al point out: misleads in precisely the way that makes the phenomena—stereotype threat—sound much more important than it is. (“Stereotype threat explains the *entire* race gap!”)

And deception is carefully balanced so as not to *technically* be a lie.

(They report that they are using an anova that controls for past test performance!)

(So editors and referees, and future lecturers, can plausibly justify accepting paper. And justify ignoring criticism?)

What are my main take aways?

Aside from articulating well some of the many feelings I (and presumably others?) have while learning social psych (creative studies+cool phenomena+non-sequitor interpretations.)

And pinpointing specific instances of fallacious conclusions.

The paper also offers a great (set of) case study(s) in common forms of motivated reasoning, used irl.

Really valuable for science imo.

What are key features of m.r. apparent here?

For one, the persuasion motive is pretty straightforward.

Strong incentive to over-sell one’s work. And one’s field.

Also apparent here:

There are obvious costs of different forms of deception. And the less costly lies are the ones we observe.

That is, twisting words and hiding disconcerting facts are harder to detect and punish than making up results or stating overt lie.

No coincidence, the former means are the way social psychologists mislead.

Also:

These less costly lies, arent just more often done.

They are *seen* as less bad. (“It’s not really lying. If read results section closely...”)

And the fallacious conclusions they lead to get internalized, more or less, as truths.

Even by those who did the original work.

And...

Despite everyone kinda knowing we do this, and knowing it’s misleading, we still do it, and even teach others to do it.

(“You gotta write your intro/conclusion so results don’t sound obvious!” “Find a way to say it that’s more surprising!”)

Summarizing what we have seen:

-Persuasion motive determines existence+direction of bias.

- Lies utilized depend on assymetry in costs.

-All aware of con. Seen as not so bad. Even prescribed.

-Conclusions internalized as truths. Even by original deceiver.

IMO this ↑ is quite general to motivated reasoning. Making this a great case study of m.r. irl.

Moving on to my second major takeaway.

A second major takeaway I have from this paper:

It’s a really good example of how perverse incentives in academia can lead an entire field toward absurd beliefs.

(cc @CT_Bergstrom)

That is, we kinda recognize that perverse incentives mess w/ academia.

But mostly we are thinking of marginal effects. Minor messing.

Like self-citations. And predatory journals.

But perverse incentives in academia don’t just make impact metrics imperfect, or bibliographies slightly bloated.

They can *even* influence our fundamental beliefs about the world, our conclusions about science. And not just one or two people. Entire fields. For generations.

Keep in mind:

We are not talking about minor papers in social psychology. Or minor wording errors. We are talking about fundamental claims. That shape the field. Forming the foundation of undergrad classes and textbooks. The central dogma of the field.

Dogmas like:

Preferences are constructed.

The situation, not personality, determines behavior.

Rather absurd claims imo. (Of course personalities exist and are meaningful! Of course preferences are not constructed on the fly!)

Dogmas that started w/ clever designs & interesting phenomena. (Good science imo.)

But turned into absurd conclusions.

By perverse incentives. Like the desire to wow. W/o getting caught lying or juking stats.

Now being taught to, and lapped up, by hundreds of thousands of freshmen each year.

B/c publication norms make it hard to criticize and correct for lies that ain’t lies. Conclusions that mislead, w/o fabricating. And make the field look informative, and their tools powerful.

Causing countless ow well motivated, truth seeking and knowledge creating academics, to devote their time and intellect to non-knowledge producing, but confusion-adding spin. And pursue results that can be so spun.

That’s the power of perverse incentives in academia+motivated reasoning. Imo.

Not just messing w/ citation metrics.

But impeding science and mislead scientists. And deeply distorting beliefs.

Eom

Addendum:

Maybe a third takeaway: is the role the perverse incentive of the “wow factor” may be playing in driving social psych to not be building up a coherent theoretical framework.

To not develop into a full fledged science to social behavior. But instead be a (admittedly valuable) list of puzzling phenomena. (And IMO absurd overgeneralizations.)

After all, a coherent theoretical lens can only be built off plausible claims.

And makes shocking conclusions less shocking.

That’s what understanding tends to do: It makes things ex post simple and clear. Not ex post “wow.”

Eom