The ACL has pushed a new reviewing agenda to build the promise of a “more consistent reviewing" -- to quote the ARR website [aclrollingreview.org]. I’m going to outline why this is not a good idea, and we should be skeptical of promoting consistency. 🧵
The idea that consistency is a bad idea for computer science conference paper reviewing is not new. Ken Church (@kchurch4) argued back in 2005 [direct.mit.edu/coli/article/3…] that we should select papers where the scores were high variance.
The reason is that papers with high variance in the scores may be of great interest to some people, even if the majority doesn't find them interesting. Also, Ken notes that conferences have a history of rejecting hugely successful papers, e.g. SIGIR rejected the PageRank paper