Thread by @annargrs

New paper📜: What Can We Do to Improve Peer Review in NLP?
arxiv.org/abs/2010.03863
with @IAugenstein

TLDR: In its current form, peer review is a poorly defined task with apples-to-oranges comparisons and unrealistic expectations. /1

Reviewers resort to heuristics such as reject-if-not-SOTA to cope with uncertainty, so the only way to change that is to reduce uncertainty. Which is at least partly doable: better paper-reviewer matching, unambiguous eval criteria, fine-grained tracks, better review forms etc /2

Which criteria and forms, exactly? Each field has to find out for itself, through iterative development and experiments. Except that in NLP such work would be hard to publish, so there are no incentives to do it - and no mechanisms to test and compare any solutions. /3

We mostly rant about peer review *after* acceptance notifications come out, but what do we expect to change without systematic work on improving its quality *between* conferences? /4

This is not to say that conference organizers are doing a bad job. But each conference makes a unique set of choices, and we have no way to tell to systematically compare and decide which policies should be kept. Also, this would be a lot of extra work. /5

Actionable steps:
- talk about peer review a lot more, build up prestige of the topic and incentives to work on it
- create new ACL roles to think about systematic testing/implementation of peer review policies, and feedback mechanism between organizers, authors and reviewers /6

Many thanks to the awesome #NLProc community. This paper would have been very different without insights of @emilymbender @yoavgo @complingy @SeeTedTalk @ani_nenkova @astent @karinv and many, many others. /7

To make this paper truly meta, it will come out in Findings of EMNLP. /8