This paper troubled me when it came out because it did not line up with my experience on a grant review panel where most reviewers (entering scores before seeing other reviews) tend to have similar scores most but not all the time. pnas.org/content/115/12… 1/
Rereading it, I noticed their sample of 25 grants that they had reviewers review had all been funded by NIH. That means those grants were in the top 20% or so of the scored grants. There were no previously poorly evaluated grants in the pool. 2/
So, yeah, duh, no surprise that there was not much consensus among reviewers of these top-rated grants as to which were the super top ones. They were missing the other 100 or so grants they would need to represent the full range of the scale. 3/
Anyway, I don't have data on what the actual rate of agreement is among NIH reviewers but would bet it is certainly better than "no agreement" as suggested by this paper. 4/

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Mara Mather

Mara Mather Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @MaraMather

29 Jun
This paper is misleading as they solicited grant applications from investigators and their entire set consisted of funded grants. So they did not have any poor quality grants (or even any that did not get funded) in their pool.
While the NIH process is not perfect, I am impressed by how carefully reviewers read proposals and know that my colleagues put a huge amount of effort into grant reviews.
Usually when I'm on a panel, I find my scores are not too different from the other reviewers, but when we disagree, we get the chance to air our disagreements in front of the rest of the room and everyone else can decide for themselves what they think.
Read 6 tweets
29 Jun
We have a broken system in science. Currently, the main bulwark still protecting science from total collapse in the US is the NIH scientific review system.

I know this may sound extreme. But hear me out. 1/N
For-profit “open-science” publishers make exponentially more money each year as they exploit the incentives of our system, in which authors are willing to pay to publish their papers and need to publish as many as possible to increase their own research impact factors. 2/
“The literature environment published in Chinese is already ruined, since hardly anyone believes them or references studies from them… Now this plague has eroded into the international medical journals.” nature.com/articles/d4158… 3/
Read 20 tweets
28 Jun
I co-reviewed for Cells with a lab member. I thought it was for the journal Cell until I submitted it at the reviewer online form. I felt duped. We gave extensive feedback and the authors responded in detail. The editor asked us to review the revision in just 3 days?!
I explained I'm working on a grant deadline and cannot do it that quickly. She was not willing to give me the extra time I needed. She told me to quickly review their response letter...
... and said, "If you don't have enough time to check the revised version, please feel free to let me know. We will ask our academic Editor to check if your concerns are all addressed." Our joint review was the only review received for this paper.
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(