Thread by @EikoFried

Dutch universities are making a move to abandon the impact factor in recognition and reward considerations. A group of 170 Dutch academics posted a critical response to this initiative. I summarize why these responses fail to convince me. 🧵

First, for context, here the initiative by @UniUtrecht we are talking about: changing rewards and recognitions. Other universities have similar initiatives.

nature.com/articles/d4158…

Here the rebuttal by 171 academics in the Netherlands, most of whom appear to be full professors. It's in NL, but google translate works well for Dutch websites.

scienceguide.nl/2021/07/nieuwe…

Their first argument: IF does not play a disproportionate role in Dutch academia; researchers obtain grants without publication in high IF journals.

The argument is supported by a story, but not by any data, which does not make this very convincing to me.

Second, they maintain IF works as intended: studies in high IF journals have more experiments, are reviewed more rigorously, thus "guarantee" higher quality.

Not only is this not supported by any data in the piece; this contrasts with data we have about IF (retraction rate higher for high IF journals). I list a few in this thread:

https://twitter.com/EikoFried/status/1415949780673499137

Authors then move on to argue that alternatives to the IF, such as narrative CVs, take longer to evaluate. They claim reviewers of grants complain this is more time consuming, refuse to review, or google hard criteria such as IF.

A list of papers and journals can ofc be helpful, but I don't think anybody is arguing against this. We are saying that the IF itself is not a very useful metric. Narrative CVs seems like a strawman here.

https://twitter.com/EikoFried/status/1415946576552415232

The authors also argue that focusing on team science "encourages further arbitrariness in the assessment" (in addition to the narrative CV). I see that: it will be difficult to properly measure team-ness, and it will be challenging to compare this across fields.

I want to add that I am also concerned that we keep broadening the list of things academics are evaluated on. But most of academia relies on collabs; I generally like the idea to encourage this (as well as good mentorship, collegiality, etc).

https://twitter.com/EikoFried/status/1410158373929111553

Then the authors say that big funders (e.g. Dutch NWO) should focus funding the best research (supposedly steered by pubs in high IF journals), whereas it is the responsibility of universities to take care of folks who teach, publish in lower IF journals, etc.

I disagree: funders have an important role in changing the academic landscape, and a lot of important #openaccess changes at universities come from changes at funders. @wellcometrust has shown how it is done, so has @NWONieuws, in part.

The authors conclude that new reward and recognition policies are just that: policies. They have little to do with science, similar to #openscience reforms generally.

Again, I really disagree with that.

Transparency, honestly, & accountability—central pillars of the open science reform—are literally enshrined in the code of conduct for Dutch researchers.

The piece ends with (translated) "Research in the medical, life and exact sciences is highly competitive and highly dependent on funding from NWO [..] We want researchers to optimize themselves for doing the best science, not for writing the best story."

I'm just going to leave that statement as it is. End 🧵

PS: follow @brembs, an incredibly knowledgeable scholar when it comes to the IF.