Thread by @dsquintana

I recently reviewed my first Registered Report (RR). It’s one thing to read about the benefits of RRs on paper, but another to see them in action.

Based on my RR reviewing experience, here’s why I think the format should be adopted at every journal (especially the fancy ones)

But first, what are RRs? In short, the RR is a relatively new format in which manuscripts are reviewed in two stages: before the data is collected and after the data is collected and analyzed cos.io/rr/

When RRs have been presented to undergrads, they often remark, “Wait, isn’t this just science?”, which is a damning assessment of our current system... Just think about how different psych would be if RRs were the gold-standard for hypothesis driven research over the past 30 yrs

The main reservation I had when I first heard of Registered Reports was that we already have trial registration, so how is this different? Well, there’s a BIG difference between posting some vague hypotheses and a detailed method

Another reservation (from other researchers) is that this stifles exploratory work. Not true, it only clearly separates hypothesis-driven (HD) and exploratory research. Both types of research are necessary in science. HD research isn’t “better”, but just often done poorly

There are several benefits with RRs (see details here cos.io/rr/) but I want to share some of the big ones that became very clear when reviewing this paper:

1. Registered reports (RRs) help eliminate the unfairness of ‘lab prestige’. Papers should be evaluated on their own merit, not by who wrote it. RRs put the focus on the METHOD. I hadn’t heard of these researchers before (maybe I’m not reading enough?) but it didn’t matter

This helps even the playing field, the only prerequisite is good science – not the name of your lab or institution. I like to think that I’m free of bias when I review, and I make EVERY effort not to be, but I’m sure the *sigh* reputation of the lab somehow plays a role

2. Registered Reports lead to better peer-review. I know this will be a citation magnet in my field (because it’s an RR on a contentious topic) so I’ve gone over and above with my review to help ensure a quality study.

Plus, with a Registered Report I can’t make any comments about the author’s methods when I review the second stage because I already approved them.

Measure twice, cut once.

This also helps eliminate gate keeping in science. A really easy way for people to shoot down a paper they don’t like is to criticize the methods, which can’t be changed after the fact. Of course, sometimes methods are plain bad but RRs solve this anyway.

3. The Registered Report format encourages researchers to plan to assess the evidence for a null model/statistical equivalence. What’s the point of a hypothesis if you can’t falsity it?

4. Registered Reports require researchers to carefully consider sample size, thus avoiding a “sample size samba” bmcmedicine.biomedcentral.com/articles/10.11…

I review a lot of papers that have really small sample sizes and it’s unrealistic to say to the authors to go and collect more data. With an RR, you can say, “you need more participants”, but this usually isn’t the case because authors know the need to justify this carefully

5. Registered Reports reduce publication bias. I doubt that if these authors get a null result & send this to a Fancy Journal as a standard paper using the EXACT same method that this would be sent out for review.

But now this paper will be published, regardless of result

To learn more about Registered Reports, have a listen to our @hertzpodcast chat with @chrisdc77 — one of my favorite episodes soundcloud.com/everything-her… Here’s a little preview