Some predictions about whether the researcher's ideology effects their likelihood of replicating a prior result. ht @jayvanbavel
First, I have no doubt that ideology CAN influence replicability. Classic Rosenthal work + more provides good basis.
So, under what conditions?
First, I have no doubt that ideology CAN influence replicability. Classic Rosenthal work + more provides good basis.
So, under what conditions?
1. Ideology may guide selection of studies to replicate. More likely to pursue implausible X because it disagrees with my priors; and pursue plausible Y because it agrees with my priors.
On balance, this may be a benefit of ideology to help with self-correction and bolstering.
On balance, this may be a benefit of ideology to help with self-correction and bolstering.
2. Ideology may shape design of studies. More likely to select design conditions to fail if I don't like the idea; more likely to select design to succeed if I like the idea.
This is a problem because of tendency for overgeneralization of limited conditions to phenomenon. But,
This is a problem because of tendency for overgeneralization of limited conditions to phenomenon. But,
it can also be a benefit for accelerating identification of boundary conditions.
The benefits are only accrued if researchers with different priors are playing in the same sandbox identifying +/- locations of those boundary conditions.
The benefits are only accrued if researchers with different priors are playing in the same sandbox identifying +/- locations of those boundary conditions.
3. Ideology may shape execution of studies. The classic Rosenthal (1966) work is most relevant for this, e.g., psycnet.apa.org/record/1967-09…
It is hard to identify any potential benefits of ideology producing experimenter effects that are beyond the explicitly defined protocol.
It is hard to identify any potential benefits of ideology producing experimenter effects that are beyond the explicitly defined protocol.
4. Ideology may shape data analysis. I am more likely to look harder at findings I don't like and identify errors in analysis, and more likely to look harder for findings I do like employing flexibility in analysis strategies to obtain them.
Also hard to identify benefits here.
Also hard to identify benefits here.
5. Ideology may shape reporting. I am more likely to write up findings I find agreeable and suppress findings I find disagreeable.
Few benefits here too. Publication bias on the ideological dimension.
Few benefits here too. Publication bias on the ideological dimension.
Solutions
5: registration of studies (no suppression of inconvenient results)
4: preregistration of analysis plans (no undetectable analysis deviations)
3: explicit, available protocols help, but ultimately independent replication
2: adversarial collaboration
1: fix needed?
5: registration of studies (no suppression of inconvenient results)
4: preregistration of analysis plans (no undetectable analysis deviations)
3: explicit, available protocols help, but ultimately independent replication
2: adversarial collaboration
1: fix needed?
