This paper troubled me when it came out because it did not line up with my experience on a grant review panel where most reviewers (entering scores before seeing other reviews) tend to have similar scores most but not all the time. pnas.org/content/115/12… 1/
Rereading it, I noticed their sample of 25 grants that they had reviewers review had all been funded by NIH. That means those grants were in the top 20% or so of the scored grants. There were no previously poorly evaluated grants in the pool. 2/
So, yeah, duh, no surprise that there was not much consensus among reviewers of these top-rated grants as to which were the super top ones. They were missing the other 100 or so grants they would need to represent the full range of the scale. 3/
Anyway, I don't have data on what the actual rate of agreement is among NIH reviewers but would bet it is certainly better than "no agreement" as suggested by this paper. 4/
• • •
Missing some Tweet in this thread? You can try to
force a refresh
