Thread by @BrianNosek

Some predictions about whether the researcher's ideology effects their likelihood of replicating a prior result. ht @jayvanbavel

First, I have no doubt that ideology CAN influence replicability. Classic Rosenthal work + more provides good basis.

So, under what conditions?

1. Ideology may guide selection of studies to replicate. More likely to pursue implausible X because it disagrees with my priors; and pursue plausible Y because it agrees with my priors.

On balance, this may be a benefit of ideology to help with self-correction and bolstering.

2. Ideology may shape design of studies. More likely to select design conditions to fail if I don't like the idea; more likely to select design to succeed if I like the idea.

This is a problem because of tendency for overgeneralization of limited conditions to phenomenon. But,

it can also be a benefit for accelerating identification of boundary conditions.

The benefits are only accrued if researchers with different priors are playing in the same sandbox identifying +/- locations of those boundary conditions.

3. Ideology may shape execution of studies. The classic Rosenthal (1966) work is most relevant for this, e.g., psycnet.apa.org/record/1967-09…

It is hard to identify any potential benefits of ideology producing experimenter effects that are beyond the explicitly defined protocol.

4. Ideology may shape data analysis. I am more likely to look harder at findings I don't like and identify errors in analysis, and more likely to look harder for findings I do like employing flexibility in analysis strategies to obtain them.

Also hard to identify benefits here.

5. Ideology may shape reporting. I am more likely to write up findings I find agreeable and suppress findings I find disagreeable.

Few benefits here too. Publication bias on the ideological dimension.

Solutions
5: registration of studies (no suppression of inconvenient results)
4: preregistration of analysis plans (no undetectable analysis deviations)
3: explicit, available protocols help, but ultimately independent replication
2: adversarial collaboration
1: fix needed?