Thread by @jrgptrs

Is Economics self-correcting? We have a new discussion paper, joint with N Fiala & @flneubauer. Quick answer is: No, rather not. We reviewed all replications published as comments in the AER. Plus, we surveyed the authors. A short thread 1/. bit.ly/3y0wMYp #EconTwitter

The AER is one of the few top econ journals that publishes comments on a regular basis. But it’s on the decline. We asked all AER-editors since 1985 what might be the reason – interesting responses, yet no explicit change in policy. 2/.

We searched volumes 2010-2020 for comments and found 56 comments on 53 original papers. Most are secondary-data based papers, some theoretical, few lab[in the field] – and no RCT. 3/.

So, is econ self-correcting? If the literature updates its prior in response to a replication, it should be reflected in the citations. We checked them on Google Scholar for both original paper and comments. Comments are barely cited. 4/.

Even papers in the Journal of Economic Literature and the Journal of Economic Perspectives do not really care about the comments. 5/.

Comments have no effect on the original paper’s citation trend 6/.

All this does not change when we account for how substantive the comment’s criticism is. Here’s when we account for own rating, but we also asked authors of both comments and original papers. 7/.

It’s also not surprising but still striking to see how little agreement there is between original authors and replicators – reflecting the sense that there’s no convincing definition or understanding in economics of what is an un/successful replication 8/.

All in all, our results suggest that economics is not self-correcting – at least not formally. This certainly relates to the fact that there’s no definition of replication success and most replication debates are quite confusing. 9/end