Thread by @profmarkreed

Just finished reading @GemmaDerrick's great new book about how impact was assessed in #REF2014: springer.com/gb/book/978331…

She's blogging soon for @fasttrackimpact but in the meantime here are my takeaways 👇

Lack of time for deliberation over conceptions of impact created a risk of group think in sub-panels that may have biased some #REF2014 outcomes (1/11)

Panelists talked about how "...over time there was a coming together", a "gut feeling of how this thing is going" as they "kind of worked out that we knew what we were measuring against." They "gained something of a common mindset" (2/11)

Many REF panelists had a broader conception of impact than the REF2014 definition (for example valuing public engagement in itself) but accepted the narrower definition they were given for the assessment (for example only valuing the benefits arising from that engagement) (3/11)

Panelists conceptions of impact tended to move from linear and instrumental pre-evaluation to no-linear and broad post-evaluation (4/11)

Non-academic user evaluators were perceived by academic panelists as primarily lending legitimacy rather than improving the quality of decisions, but may have played a "powerful antidote to overly pragmatic, simplistic groupthink-based decision making” (5/11)

Non-academic panelists wearing ties found academic jumper-wearing elitism intimidating. While some managed to bring fresh perspectives, others left the process "browbeaten by their academic counterparts" and the "hundreds of thousands of papers they had gone through" (6/11)

Evaluators suspected that some cases studies were "hyperbole", "spin", "lying a little bit", "pulling the wool over the eyes with a clever writer" to "blind" panelists with their story telling (7/11)

Senior colleagues encouraged panelists to "err on the side of positive" and "round up rather than down" as a political tool "celebrating the value of British science" (8/11)

#REF2021 panels can "work smarter" if they focus on "how to get many [different] voices aired, contributing to the deliberation and [being] considered in a balanced, democratic approach" (9/11)

My favourite quote: "impact is a sticky process and is rarely linear" (10/11)

Gemma Derrick wrote her book despite being warned (threatened?) by a colleague that "critiquing peer-review doesn't always win academic friends". Thank you Gemma for having the courage to write this book and for sharing your insights with us all (11/11)