,
22 tweets,
10 min read
Read on Twitter
We replicated 21 social science experiments in Science or Nature. We succeeded with 13. Replication effect sizes were half of originals. All materials, data, code, & reports: osf.io/pfdyw/, preprint socarxiv.org/4hmb6/, Nature Human Behavior nature.com/articles/s4156… 
Using prediction markets we found that researchers were very accurate in predicting which studies would replicate and which would not. (blue=successful replications; yellow=failed replications; x-axis=market closing price) socarxiv.org/4hmb6/ nature.com/articles/s4156… #SSRP
Design ensured 90% power to detect an effect size half as large as original study. Replications averaged 5x the sample size of originals. We obtained original materials in all but one case, and original authors provided very helpful feedback on design. socarxiv.org/4hmb6/
Eight original authors & Malcolm Macleod wrote commentaries published in Nature Human Behavior offering a variety of insights. Particularly attend to the ideas about why our results might have differed from the originals. See commentaries linked to dx.doi.org/10.1038/s41562…
For successful ones, replication effect sizes were about 3/4ths the original effect size. For failures, replication effect sizes were near 0. Could suggest influence of publication bias and false positives but does not demand that conclusion. nature.com/articles/s4156…
An outstanding example of replication fostering deeper understanding is the Shah, @m_sendhil, & @eshafir. We failed to replicate their Study 1 of 5. Then they conducted replications of all 5. They also failed on Study 1, but other parts replicated: osf.io/7z38r/
The social-behavioral sciences are undergoing a reformation. First, replication is becoming a normal activity for reducing uncertainty and improving understanding of novel claims. See @RolfZwaan et al. cambridge.org/core/journals/…
Second, social scientists are adopting preregistration to improve rigor & demarcate confirmatory and exploratory results—the latter opening possibilities but w/greater uncertainty. pnas.org/content/early/… Preregistration has doubled each year since 2012: osf.io/registries/
Preregistration and Registered Report growth is non-linear, particularly in social-behavioral sciences, and most particularly (so far) in psychology. psychologicalscience.org/observer/prere…
Third, journals are adopting transparency policies to improve reproducibility. Ex: We reviewed 33 journals that publish social-personality psych. In 2013, 0 had transparency policies. Today, 73% have some policy & 58% are relatively assertive based on TOP: cos.io/top/
Social-behavioral sciences' self-scrutiny is changing the norms for rigor and reproducibility. Change is occurring by grassroots efforts by individuals and labs, and by policy interventions by disciplinary leaders and editors. e.g., @improvingpsych improvingpsych.org
Change is also spreading. Ex: More than 100K researchers are using osf.io to register their research and make data and materials more accessible. All fields will benefit from this reformation toward greater rigor and transparency.
Vox: "What scientists learn from failed replications: how to do better science." vox.com/science-and-he…
Science: "‘Generous’ approach to replication confirms many high-profile social science findings" sciencemag.org/news/2018/08/g…
Nature: "Researchers replicated 62% of social-behaviour findings published in Science and Nature — a result matched almost exactly by a prediction market." nature.com/articles/d4158…
Washington Post: "the experiments scrutinized in this latest effort were published prior to a decision several years ago by Science, Nature and other journals to adopt new guidelines designed to increase reproducibility, in part by greater sharing of data" washingtonpost.com/news/speaking-…
Wired: "Thousands of researchers now pre-register their methodology and hypothesis before publication, to head off concerns that they’ll massage data after the fact. " wired.com/story/social-s…
NPR "'The social-behavioral sciences are in the midst of a reformation'... Scientists are..announcing in advance the hypothesis they are testing; they are making their data and computer code available so their peers can evaluate and check their results." npr.org/sections/healt…
The Atlantic: "Fortunately, there are signs of progress. The number of pre-registered experiments—in which researchers lay out all their plans beforehand to obviate the possibility of p-hacking—has been doubling every year since 2012. " theatlantic.com/science/archiv…
Science News: "‘Replication crisis’ spurs reforms in how science studies are done" sciencenews.org/blog/science-p…
BuzzFeed: "In the last four years, 125 journals, mostly in the behavioral sciences, have adopted “registered reports,” ...Similarly, more than 20,000 studies have been preregistered on the Center for Open Science’s website." buzzfeednews.com/article/stepha…
Note that most of the coverage is crediting me personally. But, like every one of these efforts, this was a true collaborative effort. Particular credit to Magnus Johannesson, Felix Holzmeister, and Anna Dreber for their leadership on this project.
