Thread by @BrianNosek

An ironic back story (h/t @Edit0r_At_Large for evoking the memory).

This paper was itself submitted as a Registered Report in 2018 but was rejected!

Reviews were excellent. Identified some limitations we could solve, others we would have needed to debate on feasibility grounds.

https://twitter.com/BrianNosek/status/1408081726044319749

@Edit0r_At_Large The journal did invite a resubmission if we wanted to try to address them. However, we ultimately decided not to resubmit because of timing. We had a grant deadline to consider.

We did incorporate reviewer suggestions that we could into the final design and proceeded.

We eventually had the full report and that was peer reviewed in the normal process.

We published the paper in Nature Human Behaviour.

The RR was originally submitted to Nature Human Behaviour.

I think the RR submission did meaningfully improve our design & odds of success.

Asked in another thread about the basis of rejection.

https://twitter.com/BrianNosek/status/1408387266414792705

https://twitter.com/BrianNosek/status/1408387700869275655

The feasibility of the stimulus set resolved well when we implemented the plan. We barely needed the back-up plans if the main strategy didn't work.

Reviewer sampling, we had a good strategy for broad outreach and expertise matching. The sample ended up quite positive about RRs.

In other survey data, we have some evidence that there is very widespread positivity about the format in these fields, so not necessarily a sampling bias based on the population.

We did more work on the power analysis based on the reviews.

And, we retained the mostly question-based, exploratory approach leaving much of the theoretical positioning for various outcome criteria in the supplemental material.

Incidentally, I think the RR format is for planned research, inclusive of confirmatory and exploratory methods.

Idea based on this experience: A tough test for RRs would be to compare standard articles that were rejected RRs with standard articles. Both are published in the conventional format, but the rejected RRs might have been improved by the review process, as ours was IMO.