Thread by @rasmus_kleis

Ask scientists about peer review, and you’ll get… a lot of things. Boundary work (the institution grounds all scholarship), appreciation (collegial quality control=more rigorous work), and many dark sides (ie it's unreliable, unfair, unpaid, unequal, and done by #reviewer2) 1/19

Let me start here-I've internalized the boundary work. I believe in peer review, for all its imperfections. I think it is among things setting science apart. I've also spend lots of time on it, including dealing with 600+ manuscripts as journal editor, based on ~1000 reviews 2/19

But while important, peer review is not the ONLY thing that define science. Many different norms and institutions together define us. As Ziman writes: “peculiarity of science is that knowledge as such is deemed to be its principle product and purpose” cambridge.org/core/books/rea… 3/19

So for example at @risj_oxford, we publish both our reports (for both practitioners and academics) and peer-reviewed in journals, books, etc (for both academics and practitioners).

While not the same, a few notes on how I think about the underlying commonality & peer review 4/19

Our research team aims to meet standards of contemporary science in terms of data+methods. But we don't always start by publishing w/academic framing and through peer review,bcs part of our mission is to connect practice & research and journal articles are not great for that 5/19

So we publish much of our work first as reports. We want them to be timely. We want them to be relevant. We want them to be accessible. (Journal publishing, which is slow, focuses on "contribution to the literature", & relies on specialized jargon, work against these goals.) 6/19

The fact these reports are often widely cited by academics I hope reflect they too - in addition to journalists & news media industry professionals - find them valuable. But while meaningful signal, citations are no direct substitute for peer review. Where does that come in? 7/19

Basically later. On the basis of the same data+research design, we then do more detailed, in-depth analysis that proceed at a journal pace, aim to contribute to the literature, and (proudly!) embrace specialized jargon of such work. Original analysis, data+design the same. 8/19

So on the basis of survey data+method behind the Digital News Report, we publish original analysis in e.g. the Journal of Communication onlinelibrary.wiley.com/doi/full/10.11… Social Media + Society journals.sagepub.com/doi/full/10.11… or New Media & Society doi.org/10.1177/146144… 9/19

On the basis of interviews and newsroom visits, we publish reports as well as articles in e.g. Journalism journals.sagepub.com/doi/abs/10.117… and Journalism Studies doi.org/10.1080/146167… or International Communication Gazette doi.org/10.1177/174804… 10/19

On the basis of data from Twitter and comScore we publish factsheets as well as articles in e.g. Political Communication tandfonline.com/doi/full/10.10… and PNAS pnas.org/content/117/46… 11/19

On the basis of content analysis we publish reports as well articles in e.g. the International Journal of Press/Politics journals.sagepub.com/doi/full/10.11…, on basis of tracking data, we publish reports and articles in e.g. New Media & Society journals.sagepub.com/doi/abs/10.117… 12/19

Point of all this (other than me being super proud of the team) is that if we didn't - from outset - work to standards of contemporary science, we wouldn't be able to publish these articles. I am proud of our reports. I am also proud data+designs meet peer-review standards. 13/19

Our reports & our peer-reviewed work is different, also share underlying commonalities: (a) primary purpose is to offer robust knowledge that help people understand substantially important things about the future of journalism (b) they meet contemporary scientific standards 14/19

Peer review is simultaneously boundary work, academia-internal status control, and an important though imperfect form of quality control. Offers predictive validity but comes with many problems including limited reliability, unfair biases & much more doi.org/10.1002 15/19

Also, let's be clear: while we don't agree on what is crap and/or irrelevant, I think most scientists would agree a lot has passed peer review, is passing peer review, and will pass peer review? Peer review important but imperfect - a human institution, inhabited by humans. 16/19

We wouldn’t accept idealized self-presentations as full account when looking at other important social institutions, and we shouldn’t pretend our own idealized self-presentations are full account either, as many have argued hup.harvard.edu/catalog.php?is… & routledge.com/States-of-Know… 17/19

Last few years, @risj_oxford we have varied btw roughly 1:1 peer-reviewed:reports to about 1:2 in our total number of publications. I’m proud of all of them, both their different qualities, and their shared standards. If we did only one kind, we'd fall short of our mission. 18/19

I see no conflict btw peer review↔️reports, public presentations, expert testimony, Twitter threads, articles for other publications, etc.

Ziman writes science aims to "enable comprehension" cambridge.org/core/books/rea…

I don't think peer review work alone is enough to do that 19/19

Sorry, proper link for this since a few DM'ed to ask asistdl.onlinelibrary.wiley.com/doi/abs/10.100…