Thread by @WendyIngman

AN INSIDER’S VIEW OF THE #NHMRC INVESTIGATOR GRANT PEER-REVIEW PANEL
I wanted to share my experience as a peer-review panel member for the recent #NHMRC Investigator grants. #MedicalResearch @NHMRC @TheASMR1 @GregHuntMP

@nhmrc @TheASMR1 @GregHuntMP I’ve been an NHMRC assessor on ECR/MCR fellowship panels for the past 4 years, this restructure is changing how we assess research excellence. We, as Australian medical researchers, are all invested in this new scheme and we need it to be successful to support our sector.

@nhmrc @TheASMR1 @GregHuntMP Before you ask, I have clarified with NHMRC and run this past my RAO @UniofAdelaide. This information does not breach my confidentiality agreement with NHMRC.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Apart from the teething problems (which are to be expected), this scheme has a lot of potential although it needs adequate funding. It puts quality over quantity and block funding means no more endless grant applications, leaving more time to do research.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide There were 5 assessors on the panel – 2 women and 3 men. Diverse in expertise. We assessed 36 applications.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide These applications were a mixture of EL1 and EL2s from applicants immediate post-PhD up to 10 years postdoc. This is the first big divergence from previous years. The old structure was ECF (<2 yrs postdoc), CDF1 (<7 years) and CDF2 (<12 years) separately.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide The applications were a mixture of Basic Science, Clinical Medicine and Public Health. I had only assessed Basic Research in previous fellowship schemes, as those 3 streams used to be separate.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I thought I would struggle with applications that weren’t biomedical, but this ended up not being a problem. Basic Science is already a hugely diverse set of applications so adding in other approaches was OK. All applications need to be written for a general scientific audience.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide The assessment criteria were purely qualitative (eg 5 = “excellent”, 6 = “outstanding”) – no quantitative benchmarking as in previous years (eg “It is expected the top 35% of applications will score 5 or higher” in 2018 ECF category descriptors).

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Impact statement was a game-changer for me. I assessed impact before looking at publications as it helped me assess the quality and contribution to science of the publications, as per category descriptor for the publication section.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide In previous years, publications was mostly about number, journal, citations, authorship position etc. But now we get to read about the impact. If the publications have been pivotal to an outstanding impact, then those publications have made a quality contribution to science.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide But how to assess impact? The category descriptors required “robust verifiable evidence” of the impact and of the contribution of the research to the impact. We were instructed that “A poorly corroborated research impact or contribution to impact should receive a score of one”.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide However we were also instructed that “The application should contain all information necessary for assessment without the need for further written or oral explanation or reference to additional documentation.”

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Here’s the problem: How do you provide this evidence? If an applicant states that their publication has XXX citations, I don’t just believe it’s true, I want to see the evidence. None of the 36 applications provided information that I would deem sufficient to verify their claims.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I didn’t think it was right to score them all a one, so I decided to try to verify them through InCites, Pubmed, Google etc. This was a really big job, the impacts were very diverse – citation metrics, awards, clinical guidelines, patents, products, policy documents.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I queried NHMRC on one of the applications I had trouble verifying. I told them I was using Google looking for evidence of the impact but had encountered a difficulty. Their advice was “take this in consideration when you mark the research”.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Once all the assessors had individually scored all applications, the scores were released to the panel. We each had the opportunity to nominate up to 2 applications for discussion at the telecon. I nominated 2 and another assessor nominated 1.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide There were 3 applications discussed at the telecon. After each had been discussed we had the opportunity to adjust our scores. Only these 3 applications could have scores adjusted, the other scores were considered already finalised.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide At the telecon, the NHMRC official said that I shouldn’t have looked for evidence of the impacts online, that we should only consider the information provided in the application. Some of the other assessors generally had also done their own checks, with varying approaches.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I asked “If I claim to have won a Nobel Prize how would you know it was true or not”? The official went to consult with their supervisor, who confirmed that we are not supposed to check outside of the application.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I am still confused about this, the category descriptor puts an emphasis on “robust verifiable evidence” so shouldn’t the impact and contribution be….. verified?

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide In checking all the evidence, I did find applicants who claimed impacts that could not be corroborated. Some were not verifiable but still might be true, in some cases though there was evidence to the contrary of what was claimed.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Every year I find a small proportion of applicants who I flag due to misleading statements. An aspect of being a peer-reviewer is to do what we can to ensure integrity of the science we assess.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I think the category descriptors rightly put the emphasis on “verifiable evidence”, but how this evidence should be presented in the application is not clear. I hope there is more guidance on this for the next round.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide The majority of applicants claimed a single impact, usually ‘Knowledge’. Some applications claimed multiple impacts across knowledge, health, etc these were difficult to assess. We were advised that multiple impacts shouldn’t be additive, but assessed holistically on balance.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide With multiple impacts, applicants often would make vague claims with insufficient detail, as there wasn’t the space in the 2000 character limit to properly explain the impacts. This might be less problematic at the Leader level where multiple impacts might fit together better.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Knowledge gain: quite a few applicants used this section to put track record components that didn’t really address the category descriptors, at the expense of explaining key elements like a well-justified hypothesis or addressing an area of critical importance.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Relative to opportunity: this is a problem. It is very difficult for assessors to score applicants with 1 year postdoc experience alongside applicants with 10 years of experience. Career trajectories are not linear and each phase has its own challenges.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide In the last 3 fellowship rounds, I assessed CDF1s for 2 years then did ECFs the following year and really struggled with changing my expectations. I had to go back and do my benchmarking from scratch so I could find the outstanding applications within the ECF applications.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide If the scheme aims to support newly graduated researchers they will need to be assessed as a group with a peer-review panel who only reads these applications.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide MD/PhDs were quite dominant in the top scores. These applications were not just in the clinical stream but also in basic science and public health. They scored well in impact, publications and leadership.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I read some applications that were in my opinion outstanding/exceptional, unfortunately none of the 36 applications were ultimately successful. There was not enough agreement in scores across the assessors, and they all ended up falling short of the funding cut off.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide At the time of assessment I didn’t know what the funding cut off would be or that there would be no normalisation between panels. Assessors need to be aware of this – the top ranked applicant within the panel doesn’t necessarily get funded.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Career disruptions: this was very prominent in the application. Of the top 5 applicants from my panel, 3 were women, 2 of whom had substantial CDs. We still have problems with this, but in the fellowship panels I have been on I feel that CDs have been taken into consideration.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide Overall, I think this is will be a good funding scheme. Yes, there are some problems that will take time to sort out. I like the impact section, and the shorter applications get right to the point – quality science contributing to improving health outcomes.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide The main problem here is that there isn’t enough money in the pool to support the Australian research community. The scheme highlights the inadequacy, but this is not a new problem. The sector is rapidly losing critical mass, our workforce has shrunk by 20% in the last 5 years.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide I support @TheASMR1 call for immediate injection of $350 million to the NHMRC as stage 1 of a stepped strategy where NHMRC+MRFF investment is increased to 3% of total health expenditure. Every $1 invested in the NHMRC returns $3.20 in health and economic benefits.

@nhmrc @TheASMR1 @GregHuntMP @UniofAdelaide That’s all from me, I hope what I’ve put here has been helpful. Interested to hear others thoughts, we’re in this together and constructive discussion is very much needed.