Thread by @bneyshabur

Some people say that one shouldn't care about publication and the quality matters. However, the job market punishes those who don’t have publications in top ML venues. I empathize with students and newcomers to ML whose good papers are not getting accepted. #ICLR2021
1/

Long thread at the risk of being judged:

I just realized that in the last 6 years, 21 of my 24 papers have been accepted to top ML conf in their FIRST submission even though the majority of them were hastily-written borderline papers (not proud of this). How is this possible?
2/

At this point, I'm convinced that this cannot be explained by a combination of luck and quality of the papers. My belief is that the current system has lots of unnecessary and sometimes harmful biases which is #unfair to new comers and anyone who is outside of the "norm".
3/

I'm going to share my views about some instances of these biases and how I have tried to protect my work against them (that is the most positive way to think about it though). These are just some instances and there is much more. I hope I could write about it at some point.
4/

My hope is that this helps:
1- Conference organizers and reviewers recognize biases and eliminate the harmful ones.
2- Students and newcomers to the field use this knowledge to be able to compete with more experienced researchers who have this unfair advantage.
5/

Writing the paper:
1- Make your paper look similar to a typical ML paper. I can't emphasize this enough. Figures and tables should follow a similar style to what is usually seen in ML. So is everything else including the abstract, introduction, phrases, organization, etc.
6/

2- Write the paper assuming the audience has a very short attention span! Many ML reviewers want to get a sense of the main results of the paper in 5 min so spend plenty of time on the abstract, first figure and contribution section. First impression matters a lot!
7/

3- Many reviewers love papers that have a combination of theory + experiments. If you are writing a theoretical paper, try to include some experiments and if you are writing an empirical paper, try to add a theoretical component.
8/

4- Cite many papers! Try to do a good job in reviewing the relevant literature AND be generous by citing many papers and giving them credit for what they have done. That is, if you are not sure, it is safer to cite.
9/

Rebuttal:
1- In your response to reviewers, be very nice to all of them even those who have attacked you unfairly. Try to explain things from your viewpoint but be careful that your response should not put the reviewer in the defensive position.
10/

2- In your response to the reviewers, try to show that you have done something extra to address the reviewer's concern (even if you disagree). This helps both by giving reviewers reasons to increase their score and AC to observe that you have addressed the reviewers concerns.
11/

3- Explicitly ask each of reviewers who gave you low scores to increase their scores at the end of your response but make sure your request is polite and respectful. Being explicit here makes a huge difference based on my experience.
12/

4- Unless your paper is guaranteed to be accepted, write to the AC explaining how you view the situation. If it seems that a reviewer doesn't understand the paper or is unreasonable, let the AC know! As an AC, I pay a lot of attention to direct messages from the authors.
13/

5- When you write to AC, think of this as pitching to AC to convince them that your work should be accepted. Try to state the main contributions and give them any information that might help them view the paper, reviews and response from your viewpoint!

6- Even when writing the rebuttal or a message to AC, please keep in mind that the reviewers and AC have limited attention span. Try to highlight your main message first, go to details next and end with a conclusion.
14/

Finally, I want to emphasize that: 1- I didn't spend much time on this so unfortunately it is likely that I have not communicated what I had in mind in the best way. 2- These are just based on my personal experience and some of them might prove to be wrong.
15/

Feel free to reply and add your experience about biases in the reviewing system in ML, how to protect your work against them, and more importantly, how to eliminate the harmful ones from the reviewing system in the first place.
16/16