Thread by @ReformedTrader

1/ Nonreplicable publications are cited more than replicable ones (Serra-Garcia, Gneezy)

"The difference does not change after publication of failure to replicate. Only 12% of post-replication citations of nonreplicable findings acknowledge the failure."

advances.sciencemag.org/content/7/21/e…

2/ "Three influential replication projects tried to systematically replicate the findings in top psychology, economics, & general science journals. In psychology (economics, Nature/Science), 39% (61%, 62%) of the experiments yielded significant findings in the replication study."

3/ "The relative effect sizes of findings that did (not) replicate were 75% (close to 0%) of the original ones.

"Prediction markets, in which experts in the field bet on results before the replication studies, showed that experts could predict which findings would replicate."

4/ "Figure 1 shows the distribution of total citation counts by the end of 2019 of the papers included in the replication projects."

5/ "The point estimate for the difference in citations is largest for papers published in Nature and Science, but the relationship between replicability and citations is not significantly different across the three replication projects.

6/ "When we include individual characteristics (# authors, rate of male authors, location/language/online implementation of experiment, publication field), the relationship is qualitatively unchanged.

"The same occurs if we control for the highest seniority level of the authors.

7/ "Moving from a relative effect size of 0 (no replication) to 1 (perfect replication of original study) is associated with 85 fewer citations.

"In replication markets, studies experts predicted would be less (more) likely to replicate have a higher (lower) citation count."

8/ "This is consistent with existing findings that experts could predict which studies would replicate.

"The relationships between market prices & replicability (and market prices & citations) remain significant after controlling for statistical power of the original paper."

9/ "Yearly citation counts reveal a gap between papers that replicated & those that did not.

"The citation gap remains even after the publication of the replication projects.

"Both results are persistent across several specifications, as we show in the Supplementary Materials."

10/ "Figure 4 shows the distribution of citations that citing papers themselves have, for each replication project, separating citing papers that cite a nonreplicable publication from those that cite a replicable one.

"The difference is not significant (P = 0.585)."

11/ "Citing papers of replicable publications are more likely to be published in a journal that is in the JCR database.

"However, conditional on being published, citing papers of replicable publications are not published in journals with a higher impact factor."

12/ "Only 12% of citations after the publication of the replication project acknowledge the replication failure.

"We conclude that the gap in citations between replicable and nonreplicable publications is not driven by new papers citing the replication failure."

13/ "Why are papers that failed to replicate cited more? A possible answer is that the review team may face a trade-off. Although they expect some results to be less robust, as shown in predictions of experts, they are willing to accept a lower expected reliability in some cases.

14/ "As a result, when the paper is more interesting, the review team may apply lower standards.

"A recent book suggests that some papers create “hype” using exaggerated and inaccurate claims. The pressure to receive grants and publish favors “…showy and ostentatious findings.”

15/ "These studies are also more likely to receive media coverage and become famous. This exposure may make the papers more likely to be cited. The book also presents evidence that the effect of the hype lingers even after a study is discredited."

https://twitter.com/ReformedTrader/status/1336398442403307521

16/ "We chose to base our analysis on three particular
replication studies because these projects had objective selection criteria (in contrast to, for example, the Many Labs projects, in which researchers selected the papers to be replicated, leading to selection concerns)."

17/ "Replication projects also typically replicated only one study out of several results presented in the original paper. For instance, in the case of the economics, only the first result of each study was replicated, even if this was not the main point of the paper."

18/ "One should be cautious when generalizing. We cannot tell whether our findings will be true for comparable papers in other fields or other journals.

"Still, taken together, current findings in the literature reveal that failure to replicate does not lead to fewer citations."