Thread by @ewzucker

In light of several “bad peer review" shared on twitter recently, I thought I’d share one of my stories. It is long (buckle up!) but I think instructive for what it reveals about the logic of the system. (It might also be a "bad author" story...)

Here goes:

More than a decade ago, a colleague & I were struggling to publish a paper we thought was pathbreaking. Versions of it were rejected by two major journals. We were frustrated we weren’t...

... getting our idea across. One thing led to another & we ended up making a massive investment over a couple of years to gut & reconstruct the paper. We felt good about the new draft to the point we were glad the prior draft has been rejected.

We then submitted it to a top...

...journal & some months later got a response: Reject. But we successfully appealed, arguing that we could address the issues brought up by the negative reviewer. We revised the paper and resubmitted it a few months later. Then we waited. And waited. And waited. After 6 months…

… the managing editor told us they were just waiting on one reviewer who promised he’d have it by the following week. A few weeks go by: still nothing. The managing editor then says that the reviewer is now promising a review the following week. Again, crickets for the next...

... few weeks. And so it went for several months. (Note to managing editors: never tell authors that a decision is about to happen because the authors will just twist in the wind till then and there could be further delays). Anyway, she finally tells us that they gave up...

... on this reviewer and solicited a new one who promised a response within the week. And then a week later, we finally get the news:
Reject

But there’s more to the story (the more interesting part)

It turns out there were three reviews. One was by the negative reviewer...

... on the first round. Now he is very positive. Publish it, he says! (We know it’s a he bc he makes no real effort to hide his identity and at some point he told us).

The other two are new reviewers (common practice in sociology in those days). One (Reviewer C) was the ...

... guy brought at the end. He was very negative. His review was of dreadful quality, someone who thinks he knows the area but doesn’t. We had a good idea who this is (in part bc we know who'd do a quick review for them) & we were dismayed they’d ask someone so unqualified...

... But it’s Reviewer B who really pissed us off. This is the one who kept on delaying & delaying, thus leading the journal to ask someone inappropriate for a quick review. But then he did send it in, around the same time. & not only did he cause problems bc he took so long...

... but the review itself was one of the most frustrating an author can receive. In short, this reviewer was quite appropriate for the paper (he’s a great scholar in my view), & he dished out some of the highest praise one sees (“As sociology, as currently constituted, the...

... analyses are much as good as it gets.”) But he damned the paper by (a) saying that we shouldn’t have drawn on a particular theorist even though we had taken what was useful and left out exactly what he had issue with; (b) saying he didn’t think we’d said anything new even...

... though he couldn’t say where it’d been said before (here’s a fun quote: “I could swear that I knew <central idea he recognizes as important> before reading the paper. But the authors want me to believe that it was reading the paper that allowed me to know that. It is...

... possible. If that is indeed the case, they have made a contribution. Certainly the authors are right that no one else has made this point explicitly.”) & (c) saying the paper’s contribution was small bc it was “normal science” and therefore “grotesquely long.” This review…

... incensed us, esp when— after reading the review several times, I realized who he was— a very prominent scholar who was known to have the ear of the editor & could easily tip the balance in a split decision. It seemed clear that it was bc of his status that they’d waited so…

... long for him to produce his (bullsh**t) review. How did I know it was him? Well it just so happened that I had had extensive correspondence with this dude over the prior few months, and there was a favorite phrase of his in the review. Once I saw that, it was obvious that..

... the whole review was written in his inimitable style. What’s more, I was already pissed with this dude for what I considered unprofessional behavior that led me to waste a great deal of time on something. So once I realized it was him, I was infuriated. I figured there...

... was a pattern here & his motives were impure: he was (perhaps unwittingly) trying to sabotage me!

What to do? Well, we did something very unusual and questionable: emailed the dude & gave him a piece of our minds!

Yep, this is not something we teach in grad school...

... and it’s something I’d certainly discourage myself. In fact, my experience in this case may illuminate *why* this isn’t a good idea.

So that’s why I’m writing this: to shed light on what happens when you think you know who a sh*tty reviewer is and you write to him...

... to give him a piece of your mind. I’m sure many authors have fantasized about doing this. (Will be interesting if others offer related stories.) So what happens when you try this? Well, I can only tell you what happened in this particular case (& only from my perspective)...

... In short, our approach was basically to write a collegial rebuttal to his review, pushing him to defend the weak criticisms by which he’d damned our paper.

The note's subject line: “Your reading for the weekend: why <name of offending theorist> deserves a second chance”...

...The note addressed him directly (“Dear <name of reviewer>”) and went straight into rebuttal, as if we had been having a conversation about our paper. In particular, it began by laying out why he was wrong to tar our model with the errors of others who’ve built on the...

... theorist in question and why it made good sense for us to take the “baby” from that theorist and throw out the “bath water.” We also rebutted the other points. On his wondering about whether our idea was really novel, here’s what I wrote: “I would really take issue with…

... one thing you said: that the value of the contribution can be questioned because you somehow knew it before and it was already implicit in the literature, if not explicit. There are two problems with this, one minor and one major. The minor problem is that implicit or...

...unstated arguments don’t count for squat. (And in any case, where is this implicit? I find no precedent for it.) I have no doubt that most smart sociologists have many useful papers they could write if they had the time, and those papers anticipate those that do eventually...

... come out. But that is hardly a reason to discount the contribution of those who made the effort of (developing and) articulating the idea so that lesser minds can benefit too. The major problem is that, as you well know, this complaint can be made about any contribution to...

… social theory. We are theorizing about theorizing agents, and so the chance of us saying something truly new is slim to none. Our role is simply to articulate social truths in a more disciplined way and thereby to help separate fact from fiction. In fact (and I’m sure…

… someone has said this), I would argue that insofar as our theories are successful, *it must be the case* that our readers already intuit those theories, or at least that the theories are *compatible with* the reader’s intuition. So in short, ‘not surprising to <Reviewer B>'…

... is not an appropriate criterion for rejecting a paper. How many (even very good) papers can meet that criterion?”

Over the course of the note, I also noted that we didn’t think his review was a bad one but essentially incredibly niggling and that it did particular damage…

... bc of the delay and the additional, poor-quality, review that came in as a result. We also noted the names of all three reviewers, and said “no, no one told us this. But sometimes you don’t need networks to figure things out.”

Finally, I ended the email by telling him...

... we had (again) appealed the decision to the editor, and said that “The question for you to consider is whether, after reading this email and seeing how the decision was arrived at, you are OK with the decision to keep it out of <name of journal>. If you are OK with it, ...

of course do nothing. You certainly owe us nothing in this regard. But if you now think that a paper that should be in the <journal> will not be, a note to <editor> would obviously help that cause and be much appreciated (Of course, we prefer that you not mention this note....

... But we will understand if you think it needs to be mentioned). Let your sociological conscience be your guide.”

(The note was signed w my name only because...…... my coauthor didn't have a relationship w him)

Let me pause here for a moment. Did we really expect...

that he’d contact the journal and say he/d rethought his review? Naively, I guess we did think there was a chance of that. Basically, since we thought he couldn’t defend any of his points, he’d realize he had no other choice: he’d contributed to the rejection of a worthy…

paper for no defensible reason!

But then there was that matter of what we’d say to the editor. Problematic, right? I’ll come back to that later.

So, did he respond, and how?

Perhaps surprisingly, he responded the same day and pretty much along the lines we were hoping. The response started with “Dear Ezra, I should have known it was your paper. I was the delayed reviewer. And so I believe delays were my fault.” He then said that it was an...

... “enormously frustrating” paper for him to read (didn’t say why) and said that his recommendation was to “cut [a particular part of our paper], force the author to cut more, and then publish.” He also said that “the newness of the <core idea> is one I thought about a lot…

... and I concluded that it is implicit in much of the thinking I have -- reflected in the work of my students, <name>, and also in the work of <name>, on <name of project>. So I was aware of the possibility that it was not new to me, and then also aware that it might be new…

... and then concluded as I wrote that even if it is not new it has never been so clearly articulated, and so was a contribution leaving aside the possibility that it was a totally new. You are undoubtedly offended by my conclusion that the paper is a normal science paper,…

... but as I read it, that is what it was, and I think the comment was fine. Especially since the < journal> publishes normal science routinely, it certainly was not a fatal comment.”
(I hadn't mentioned the 'normal science' line in my note to him so this was a bit a tell)...

... He then went on to say that “Reading my review now I feel it was pretty fair: 1. seemed to understand the point that was made. 2. appreciated the analytical introduction as the core of the paper. 3. felt that the <part of paper was> unnecessary to the theory point.”...

... He concluded that “the issue of contention then does boil down to whether or not I am fair with respect to banging you for building around <the literature that built on the offending social theorist> I will give that some thought."…

... He concluded by emphasizing that he valued my "friendship and esteem."
My coauthor and I then each responded with four main points:
(a) since the editor told us that only one reviewer recommended acceptance, was he sure that he checked the right box on the form?

(b) A challenge to him to show where his students’ work anticipated our paper in any way; (c) Suggesting that maybe what frustrated him was his perception that we were building on a sh**y literature but we’d now shown he was wrong about that; and...

(d) that there was a contradiction between two of his assertions: (i) that while the core idea of the paper is appealing, it was anticipated by his favorite scholars; and (ii) that the core idea of the paper is unappealing because it just fixes a problem in a crappy literature...

... There were more twists and turns to the correspondence. At some point, I pointed out that the ‘normal science’ comment was effectively the way an academic snob kills a paper; sure, any journal publishes a lot of ‘normal science’ but no one *seeks* to publish normal science…

… But let me cut to the chase(!).
Several days after the initial email, the reviewer sent a concession email:

“I wrote a shitty review of a paper in a rush because it was late because I am doing too much and you believe it was rejected from <journal> because of it…

… I don’t believe it was rejected because of that. I will agree with all the rest.”

Ok, so he had conceded!! But did he write the editor?

No. He said he was willing to do so (despite asserting that he hadn’t made the difference)…

… But he asked us a question that threw us back on our heels:
If the editor were to ask him whether he had spoken to us, would we back him up in saying that we hadn’t?
You’ll note that we originally suggested he keep it quiet, but said it was up to him. But now he said he...

was willing to tell the editor he had thought better of his review, but he didn’t want to tell the editor (if the editor asked) that he was prodded to do so by the authors. But he needed us to keep it quiet as well, else he would be caught in a lie. What to do?

We thought about this long and hard. This was our last, best chance. This dude had the ear of the editor. And was it wrong to hide our correspondence from the editor? All we were doing was correcting the process, by showing that the reviewer couldn’t defend his critique!

...

... Well, we decided that it *was* wrong. Our considerations were a mix of strategic and ethical. In short, we realized we would essentially be entering into a conspiracy, & with someone we didn’t fully trust. And not only was this wrong, but who knows where it would lead…

... It seemed like the kind of lie that could eventually become a mountain of lies. And so we realized that we were like the proverbial dog that had caught the car: we had gotten a reviewer to concede he’d written a crappy review, but it wasn’t clear what we could do with it...

... So what *did* we do? The only thing we thought we could do: We wrote an email to the editor, attaching the initial note from us to the reviewer. We explained our motives for what we did and discussed the end of our original email as follows:

“A last note is that you can tell from the parenthetical remark at the end that while we say we were fine with <reviewer>’s mentioning our email to you, we were less than enthusiastic about it. But now we realize that that is the best course. We have nothing to hide. As we…

… said in a previous email, we are merely <journal> authors who think that errors in the process led our paper to get less than it deserved. We think that if you ask <reviewer> his opinion once again, he will tell you that the paper indeed deserves to be published. But now it..

…is now up to you to confer with <reviewer>, and take his reaction under consideration as you deem appropriate. (Note finally that the main reason to reconsider the review has nothing to do with <reviewer>. It’s that the third review was atrocious. You might ask…

... <Reviewer B’s> opinion on that too.).”

The editor responded with a note, definitively closing closing the case.

Here are a few key lines:

“The project of a disciplinary journal can go forward only under certain arbitrary rules. One of those is that the identity of…

... reviewers - however apparently obvious it may be - is never actually named publicly by anyone in the system. That rule has been breached here, and a separate conversation has begun, completing the triad, so to speak. The result has been the airing of a lot of things, of…

... various kinds, and with various implications. Some of them concern the instant case, others concern the journal's processes, content, and behavior, still others involve characters, motives, and other personality details of the persons involved.…”

...

“The journal can function as a disciplinary project only via our common adherence to the belief - true or false in fact, but necessary as belief - that authors, reviewers, and editors conduct their business in good faith. Any complaints about good faith are directed to the…

… editor, as the person who - because he stands between author and reviewers - is ultimately responsible for each to the other.”

“But much of what is transpiring in this correspondence at this point is precisely what the fiction of double-blind review was designed to avoid…

...point by point consideration of this and that in a review between reviewer and author, speculations about journal policy, attributions of character and motives, and so on. These are things which the double-blind system was designed to keep where they belong - in the personal…

… world or, in a few extreme cases, in complaints to the editor. Much of what happens in the publication system is structurally given - as I don't need to tell any participant in this conversation - right down to and including the emotions involved about…

… all of this structurally generated anger because, in fact, it is the only way we can live with each other given that we are all ambitious, egotistical minds who view our own ideas in a much more favorable light than we give to others, and who indeed think that the class of…

… ‘our own ideas’ includes many things which, as it happens, many others think are ‘their own ideas.’ If we actually told each other what we genuinely think in our hearts of hearts, academia would not last very long. Double-blind reviewing - and the fiction that it is indeed…

… double blind - is how we avoid this catastrophe.”

“I have reviewed the instant case. I am confident the journal followed its policies properly, and that being the case I am not going to rethink the judgment, nor am I going to gloss the judgment beyond what gloss has already…

… been given. I am not going to read the paper and somehow make ‘my own’ judgment. That is precisely the form of arbitrary editorial decision that reviewers and boards are designed to avoid, and I am not going to undercut my colleagues' and my collective judgment because I do…

… or do not think something about a paper myself. Much that appears in <journal> does not fit my personal standards and desires. That's not a reason for not publishing it; the journal is not a personal journal.”

I will close this with four observations.

First, while there is great wisdom in the editor’s response, I think many will find the last paragraph a bit curious. In short, reasonable scholars can disagree with the premise that the editor is just a referee who doesn’t play the role of peer him/herself. Personally, my...

... main criticism of this particular editor over the years of his tenure was that he didn’t actually read the papers and weigh in on them.

Second, the editor nicely articulates a very deep point, one that is hard to see outside of cases like this. In short, “blindness” in…

... the peer review system is about more than trying to avoid bias created by knowing the identity of the scholars involved (indeed, the latter is not even attempted in a single-blind system); rather, it is about not *relating to them on the basis* of this identity. And that...

... in turn puts the system in tension with the logic of collegial relationships by which we generally wrestle with ideas. Personally, I find academic peer review frustrating for many reasons. And one of them is that it actively works against effective scholarly discourse...

... that can only be achieved through direct dialogue. I don’t have a great resolution for this tension (other than post-publication commentary and reply, though I have 'instructive' stories about that too). I just think that players in this system should be aware that...

... insofar as peer review works, it is largely *despite* limiting effective dialogue.

Third, while I wrote in my original note to the reviewer that “sometimes you don’t need networks to figure things out,” I now see this is a problematic statement.

Basically, the…

... whole story wouldn't have happened if the dramatis personae didn’t know each other well. And one could argue that anything we learned from this process is a kind of insider knowledge inaccessible to those who are more on the periphery. (another reason I’m writing this up)..

… I could argue that we weren’t asking for special favors but only the consideration our paper deserved; & I’d note that our appeals ultimately failed. But still.

Finally, I don’t regret what we did. To the contrary, I was glad to have verified that our sense…

…of how the paper was (mis)treated was on-target and it gave me useful perspective on how top scholars go about their work. I’m also glad we had the wisdom not to accept an invitation to conspiracy (which we were to blame for encouraging). Was good to have faced that test. /END