The ACL has pushed a new reviewing agenda to build the promise of a “more consistent reviewing" -- to quote the ARR website [aclrollingreview.org]. I’m going to outline why this is not a good idea, and we should be skeptical of promoting consistency. 🧵
The idea that consistency is a bad idea for computer science conference paper reviewing is not new. Ken Church (@kchurch4) argued back in 2005 [direct.mit.edu/coli/article/3…] that we should select papers where the scores were high variance.