Thread by @BrianNosek

Happy to elaborate. Think of preregistration of analysis plans as hypothesizing, data analysis, and scenario planning all rolled into one and without knowing what the data are. This creates a novel decision-making situation. 1/

https://twitter.com/Maxyusc/status/1132226934690029568

For example, the first time preregistering an analysis plan, many people report being shocked at how hard it is without seeing the data. It produces a recognition that our analysis decision-making (and hypothesizing) had been much more data contingent than we realized. 2/

Without the data, there is a lot of new mental work to articulate precisely what the hypothesis is and how the data could be used to evaluate that hypothesis. My odd experience was believing that I had been doing that all along, w/out realizing that I used so much discretion. 3/

Just that is hard enough. Then comes the scenario planning. What would you do if your assumptions were violated? Is your plan robust to other data patterns? Have you thought of all the reasons you might want to exclude data?

I don't know. No. No. Are common answers. 4/

It takes time and experience to get good at anticipating these. Over iterations, one can develop Standard Operating Procedures--reusable rules for exclusions, assumption violations, data collection challenges, and other realities that were unexpected. 5/

After data acquisition and analysis begins, it is common to realize that the preregistration was not clear enough about the analysis plan. "Did I intend to transform the raw data or the means? My plan is vague." Oh, man! It takes some practice to be clear and precise. 6/

Almost inevitably, there will be deviations from the plan. Not every scenario was anticipated. But, now that I committed, I feel nervous about change: "Am I exploiting discretion? Am I damaging diagnosticity of inference?" Maybe, maybe not. It takes practice to know. 7/

It also takes practice to become confident in making those deviations and preparing to justify them in reporting the results. When writing those results, what do you include? It is hard to keep track of the plan with the narrative demands of writing a paper. 8/

It is hard to decide how to disclose the deviations to be useful for the reader, deal with space constraints, and portray effectively what I did. And, even with having the preregistration, it can be hard to keep track of it in writing, particularly in collaborative work. 9/

Upshot: Preregistration is hard. It is also exciting, confidence building, and evidence bolstering.

And, even better, lots of people have struggled through it and provide some tools to accelerate the learning curve for others. 10/

For example, try this guided workflow supporting preregistration: osf.io/prereg

Read this overview paper: pnas.org/content/115/11…

Read about SOPs: stat.berkeley.edu/~winston/sop-s…

Overview page: cos.io/prereg/

Those are just taster support docs. Good luck! end/

Postscript: Embrace the aphorism "Perfect is the enemy of the good."

With all this, it can feel daunting to start with preregistration. Just remember, as with any skill, it takes practice. And, practice pays off.