Thread by @adamdoupe

As I was filling out my review preferences for @USENIXSecurity , I realized that PhD students don’t have an idea of how this works.

So let me explain
/thread

I’ve had arguments in the past with coauthors and collaborators about the value of a paper’s title, abstract, or even those “categories” you select when submitting the paper.

Since being on the inside now, I realize the importance.

So, here’s what I had to do for @USENIXSecurity : go through all the roughly 500 papers submitted and give them a relative reviewer preference of -20 to 20

(The absolute scores themselves don’t matter, as they are normalized by HotCRP)

One note: this is my personal process, others may vary significantly

So what’s the goal here? My goal is to review papers that are in my area, so that I can contribute a meaningful review

Personally, I am also looking for interesting papers. When reading a stack of 20 papers, an interesting one can be a breath of fresh air

HotCRP has a nice feature that allows reviewers to put in their preferences based on the conference categories

The HotCRP will then assign a score to each paper on what it thinks your reviewer preference will be based on your stated preferences and the choices the authors made for categories

I always sort the massive paper list by this value, it’s not always perfect but it’s a good approximation (and a good starting place

Depending on how much time I have I may use the view that shows me paper titles and abstracts, or just the paper titles

I then read through all the titles and enter a score for each

Sometimes I click through to the PDF to check out the paper further, but this takes a lot of time and is only for (1) really interesting papers or (2) papers I think I’ve reviewed before

(What to put for reviewer preferences on papers already reviewed is the subject for another tweet storm)

Essentially, everyone on the PC is going through all the papers titles and abstracts putting in preferences of what papers to write

It takes a long time, so help us out

What this all means is that your paper’s title and abstract directly influence who bids on your paper, and it behooves you to attract competent reviewers.

One cool development is that some conference are starting to do NLP to learn the reviewer’s papers and match submitted papers to them

has done this in the past and I was very happy with the results

/end