[Thread on reviewing for #MachineLearning confs, after receiving the reminder from @iclr_conf (#ICLR)]
Posting response to @iclr_conf's request for reviewing here in the hope (again) that we can change the reviewing structure of ML conferences to promote better science.
1/
Posting response to @iclr_conf's request for reviewing here in the hope (again) that we can change the reviewing structure of ML conferences to promote better science.
1/
Whether it is @NeurIPSConf (#neurips), @icmlconf (#icml), @iclr_conf (#ICLR), @RealAAAI (#aaai) or any other crowded ML conf, the reviewing structure that involves a fixed review window, multiple assigned papers, unlimited supplemental material, etc., promotes the following:
2/
2/
a) Reviews are handed off to inexperienced reviewers, directly or indirectly, due to the time pressure involved.
b) It leaves reviewers with little time to digest the technical details, leading them to potentially miss subtle errors or undervalue the merits of the work.
3/
b) It leaves reviewers with little time to digest the technical details, leading them to potentially miss subtle errors or undervalue the merits of the work.
3/
c) It leaves reviewers with even less time to write insightful reviews that can help advance the science.
Couple that with the fact we don't have shepherding, which means reviewers are forced to either (a) reject close-to-good papers or (b) accept not-quite-there papers.
4/
Couple that with the fact we don't have shepherding, which means reviewers are forced to either (a) reject close-to-good papers or (b) accept not-quite-there papers.
4/
Now to Exhibit 'Z', using @iclr_conf invite as an example (don't meant to pick on ICLR; it's the same with all ML conferences).
a) I will be assigned 3-6 papers.
b) I will have 16 calendar days for reviewing them.
5/
a) I will be assigned 3-6 papers.
b) I will have 16 calendar days for reviewing them.
5/
All the while, remember that:
a) I will still be teaching a decent-sized undergraduate class, with possibly no TA support due to COVID-19 budget cuts.
b) I will be working from home, and will also be trying to survive a pandemic, advise students, manage parenting, etc.
6/
a) I will still be teaching a decent-sized undergraduate class, with possibly no TA support due to COVID-19 budget cuts.
b) I will be working from home, and will also be trying to survive a pandemic, advise students, manage parenting, etc.
6/
So no, unless I want to do a particularly bad job of being a reviewer, I cannot accept this invite. It's the same reason I did not accept @NeurIPSConf (#neurips2020) review request this year either (which gave TWO MORE DAYS than ICLR!!!👏🏾).
7/
7/
My role as a scientist is to help advance science. I don't want to be a gatekeeper to good science, but I also don't want bad science to go through me. And I cannot, in good conscience, accept these reviewing requests because of the absurdity of what is expected of me.
8/
8/
Yes, there are people out there who are remarkable in their abilities to manage all of this. But how many of those folks are out there really? I am hoping there are more mere-mortals like me? Or perhaps I am uniquely incompetent? You can decide for yourself.
9/
9/
If #MachineLearning conferences really care about advancing science in a fair manner then a solution is some sort of a rolling review process, much like journals. The publication model will also change for conferences (perhaps something like an associated journal).
10/
10/
I am of course not an influencer within the ML community and perhaps this thread will not even be read by many. But if you are in a position of influence in these #MachineLearning conferences then please, for the love of science, do something about it.
11/11 [End of Thread]
11/11 [End of Thread]
@threadreaderapp unroll
