Thread by @ShawnMcGuirk

Join me @NSERC_CRSNG & our fantastic panelists this morning at 8:30AM at #CSPC2020, to discuss how the concept of research excellence is changing, around the world!

@AnnaHscientist moderates the panel featuring @StephenPinfield Erika Kraemer-Mbula @LinZhang1117 @LeonievanDrooge

https://twitter.com/sciencepolicy/status/1329039212373544963

. @AnnaHscientist launched the #CSPC2020 panel with a description of the mission of the @DORA (sfdora.org), and shared a recently published resource on rethinking research assessment, on unintended and cognitive biases in research assessment: sfdora.org/wp-content/upl…

. @LeonievanDrooge conducted a review of 30 yrs of Dutch #ResearchExcellence policy w/ Rathenau Instituut (🔗 rathenau.nl/en/knowledge-e…), which raised many fundamental questions.

If we fund excellent researchers, how do we describe the rest of the community? “Not excellent”? “Not yet excellent”? Should we aim for a zero-sum game?

This creates pressure to prove one’s value, & adds barriers leading to silos & poor distribution of funding

Alternatively, we can think of excellence as being a threshold – anyone can be excellent if they reach a certain standard, regardless of the performance of others.

Both of these approaches can be (and are) used.

What is less clear is what fits within the notions of excellence and quality. Publications are the gold standard, but are far from the only way researchers show impact.

Dutch funders have taken a new approach, “Room for everyone’s talent (🔗 vsnu.nl/files/document…

This framework redesigns action along 2 lines: redesigning academic career paths, and assessing the quality of research and research proposals ⬇️

See also:

The new research evaluation protocol of the Netherlands, that addresses research culture : 🔗 vsnu.nl/files/document…

Developments in the Netherlands and the move away from impact factors: 🔗 scienceguide.nl/2019/11/dutch-…
'

. @LinZhang1117 presents an overview of recent changes in research assessment practices in China, notably a farewell to the use of science citation index, a shift from publication-based assessment to more comprehensive peer review, and new priority towards local relevance

1 in 3 publications from Chinese researchers must now be in domestic journals.

@LinZhang1117 's recent paper goes into more detail, on challenges towards implementation of these changes: 🔗 scholarlyassessmentreports.org/articles/10.29…

She further notes that it is important to find a balance and fund a diversity of research, across: disciplines, types of research, development stage, global and/or local relevance, strengths and career stage of researchers, and individuals versus groups of researchers.

Commercial & political interests can also influence how research funding is assessed and allocated, and national interests can conflict with the desire to collaborate more broadly, with different partners or with international experts.

Erika Kraemer-Mbula @go2uj recently published a collection with the @IDRC_CRDI, on Transforming Research Excellence: New Ideas from the Global South (🔗 africanminds.co.za/transforming-r…). She notes that excellence is an essentially contested concept, which shouldn’t be taken for granted

#ResearchExcellence also can't be uniquely defined, it is pluralistic. Therefore it is critical to be precise and transparent, with assessment tailored to the context in which the research proposal exists. Different contexts can create tensions, however:

To move forward, it is important to mobilize all actors and platforms, and focus efforts on strengthening research ecosystems as a whole … do not create islands of excellence that are disconnected from each other.

. @StephenPinfield @RoRInstitute describes major ways the concept of excellence is being challenged. (1) it is not a unique concept, needs to be pluralized to a wide range of metrics and quality. (2) the concept has lost its value and needs to be replaced (by what? still unclear)

What we need is data & experimentation, so that funders can make evidence-based decisions about their assessment processes. This is what @RoRInstitute hopes to achieve, with a large consortium of strategic partners (🔗 researchonresearch.org) including @CIHR_IRSC in 🇨🇦.

This includes a literature review, looking at multiple factors. Preliminary findings:

RoRI is also running a first wave of projects, incl 5 w/ funder partners on (1) criteria, (2) excellence, (3) FAIRware, (4) pathways, and (5) randomisation. Also working w/ publishers on three others projects, on (1) peer review, (2) diversity, and (3) alignment of incentives.

Want to continue the conversation? Join us next week (23 - 27 November) at the virtual conference on ‘Responsible Research Assessment’ organized by @grc_research @UKRI_News @NRF_News. Registration is free and open to the public: eur.cvent.me/NV1Mv.

Check out also the working paper prepared by @RoRInstitute, @AnnaHscientist and collaborators (with minor contributions from me!), published today on the changing role of funders in responsible research assessment: progress, obstacles and the way ahead 🔗 rori.figshare.com/articles/repor…

@threadreaderapp unroll please!