Thread by @GordPennycook

New working paper! "On the belief that beliefs should change according to evidence: Implications for conspiratorial, moral, paranormal, political, religious, and science beliefs" psyarxiv.com/a7k96

Read this thread if you're curious about how I (almost) fucked this one up.

First, I'll briefly explain the key finding.

In essence, we show that "actively open-minded thinking about evidence" (AOT-E) - that is, self-reporting that you think beliefs and opinions *should* change according to evidence - is a really strong correlate of lots of things.

IMO, that's cool and important (read the paper if you're interested)... but let's put that aside and focus on how I almost fucked this one up.

I have been sitting on Study 1 of this paper since grad school (we ran it in early 2016). The correlations were *ridiculous*. In a MTurk sample (N = 375), AOT-E predicted a set of conservative political opinions at r = -.61 and a set of pro-science beliefs at r = .65.

According to this paper by Gignac & Szodorai (psycnet.apa.org/record/2016-41…), an r of .45 is in the 95% percentile in social psych. So r's > .60 is, like, super high.

Furthermore, @DavidFunder & Ozer recently published an excellent paper where they argue that "very large effect sizes" of r = .40+ are likely to be "gross overestimate(s) that will rarely be found in a large sample or in a
replication"

https://twitter.com/DavidFunder/status/1126226285670223872

So how did I almost fuck up? It's more mundane than perhaps you're thinking:

I almost published the paper without first figuring out why our effect sizes were so big. Our conclusion was almost "conservatives hate revising their beliefs according to evidence". Wrong! (sort of)

The key paper that set us on the correct path was from Stanovich & Toplak. Using a similar scale (& some of the same items) as in our AOT-E, they found that they (who originally constructed it) incidentally injected a bias in the scale.

https://twitter.com/JonHaidt/status/1106891588846977024

Specifically, the scale asks about revising "beliefs" according to evidence.

For religious people, in particular, this sounds like "religious beliefs". When the scale is de-biased, the correlation with religiosity decreases markedly. (religious people are still lower in AOT)

So, following Stanovich & Toplak, we changed our scale to ask about "opinions" instead of "beliefs". This decreased our effect size estimate down from the insanely high r's of ~.60 (religious belief being a covariate of both conservative opinions & anti-science beliefs).

But this wasn't the only issue. The second problem was that we used Mechanical Turk, which skews very strongly liberal.

When we looked at a more representative sample, Lucid, we discovered that our correlations were being carried largely by Democrats (overrepresented on MTurk)

Although the revised ("opinion") AOT-E is somewhat predictive among Republicans (e.g., r = .19 with pro-science beliefs), this is a far far cry from the r of .65 in the first study.

And even though AOT-E is correlated with liberalism (in the aggregate) in our more representative Lucid sample, it does not reliably (or strongly) predict political opinions *among* conservatives.

Thus, our conclusion is very different from "conservatives hate evidence". In fact, we essentially have no idea why the AOT-E scale is relatively poor at predicting beliefs among conservatives.

(suggestions are welcome! see the paper for some speculation)

The resulting paper paints a more nuanced & uncertain picture than what Study 1 (for about 3 years) led me to believe.

But it's also not completely wrong. Which is what it would have been if I didn't bring some skepticism to the huge effect sizes that we initially found!

Thanks for reading!