Thread by @LeonievanDrooge

New #evaluation protocol for academic research published in the Netherlands: #StrategyEvaluationProtocol #SEP 2021-2027 by @deVSNU @NWONews and @_KNAW. Something old, something new. A thread. vsnu.nl/files/document…

The goal of #SEP remains to maintain and improve quality and societal relevance of research. A unit is assessed in light of its own aims and strategy; and aim and strategy depend on institutional context, research field, nature of unit etc. Nothing really new.

A #SEP evaluation doesn’t stand alone; it is a six-yearly event in the ongoing quality assurance cycle. Not really new, but SEP evaluations are far more explicitly positioned as part of a continuous quality assurance process.

The name is new: it is now Strategy Evaluation Protocol, not Standard Evaluation Protocol. This is to stress the importance of and attention for the strategy of the research unit that is evaluated. #SEP

Evaluation process #SEP remains the same: board of uni/research institute is responsible, also for Terms of Reference, follow up; research unit presents self evaluation report and organizes site visit; assessment committee writes a clear report with recommendations

Three assessment criteria of #SEP remain the same: (1) research quality; (2) societal relevance; (3) viability

Four specific aspects differ from previous protocol: (1) Open Science; (2) PhD policy and training; (3) Academic culture (openness, social safety, inclusivity); (4) HR policy (diversity, talent management) #SEP

New is that the assessment report should consist sharp, discerning texts and clear arguments ONLY. In other words: no more use of prescribed and discreet assessment categories (5-1 or 1-4 or excellent-poor or otherwise) #SEP
rathenau.nl/nl/wetenschap-…

There are no prescribed indicators (nothing new). Protocol contains a list of example indicators, and stresses that an indicator should relate to (aim and strategy of) a research unit.

However, this is new and in line with several initiatives and developments, it is stated that the use of #JIF is not allowed and that the use of H-index is strongly discouraged. #SEP

The self evaluation #SEP report is max 20 pages, excluding tables and case studies (new, used to be 15 pages) and presents aims, strategy and results in a coherent and narrative argument (new: attention for the narrative argument for the self evaluation report).

Case studies #SEP (previously: narrative) relate to a project, programme or unit. They are about academic and societal aspects and about the connection between the two; this connection is seen as essential in many academic domains and disciplines. There is no prescribed format.

New #SEP is in line with, or enables, or defines, current developments such as #OpenScience @DORAssessment Recognition & Reward etc

For those unfamiliar with Dutch #SEP: there is no national agency, there are no national block funding decisions being made, there is no benchmarking, aim is not to compare. Aim is quality assurance: maintain quality and improve if necessary. It is unlike UK #REF2021.