Thread by @MrJaniR

Why is it that I spent some three hours over the weekend scrutinizing a #systematicreview in minute detail? I hadn't expected needing to explain my motives but someone forced my hand so here goes. This will be a shorter thread. 1/10

https://twitter.com/MrJaniR/status/1462670746585604098

When someone asked Sir Edmund Hillary why he wants to climb Everest, he responded: “Because it’s there”. Similarly, I examine these reviews because I know I can do it well and because I don’t see anyone else doing it out in the open. 2/10

I used to do this for a living. For nine years I worked as Managing Editor of Cochrane Work, which means that not only did I peer referee countless reviews, I led our group’s peer refereeing process. So, I taught our authors and our editors and peer referees. 3/10

To people not familiar with academia, peer review (or refereeing in @cochranecollab bc easy to mix reviewing w conducting rev) means quality control. In practice it is a process of finding problems in submitted manuscripts so that the authors can fix them before publication. 4/10

Sometimes it may be worthwhile to do this even after publication. I think this is very much the case with these reviews that have massive public health relevance, such as ones about effectiveness of face masks. 5/10

To the untrained eye picking faults may appear malicious. But faults in a review does not equal faults in the authors! That is a gross misunderstanding. If anything, it is an act of kindness to show where improvements are needed. 6/10

In any case, picking faults is possible only if there are faults for one to pick! I use a systematic process with a validated checklist (AMSTAR2), basically peer refereeing every review from beginning to end just like I would do if a journal asked me for my opinion. 7/10

And if the journal or the authors do not see the benefit (they are under no obligation to respond to random Twitter comments) then readers of my commentary can hopefully interpret the review I’ve examined and its results and conclusions in a different light. 8/10

Thing is, systematic reviews and especially ones with meta-analyses have, by default, ingrained in them an aura of mystical everlasting truth. It takes training and expertise to see past the shiny surface and examine the innards. I’m offering mine as a shortcut of sorts. 9/10

I welcome anyone to challenge my interpretations and judgments by using rational arguments. Something like: “Your mother was ugly when she had you” isn’t quite up to par. We’re all on the same team in the end so let’s play nicely, eh? END 10/