Thread by @tdienlin

Okay, #ica20, let's do this. Here are my thoughts on #openscience and #opencomm for the more qualitative approaches in our field!

(Beware, long thread.)
(Also, please be kind.)

>Background: I was VERY curious how our agenda for #opencomm would be received in our field (if you don't know what I'm talking about, here's the link: academic.oup.com/joc/advance-ar…).

>Basically, we're calling for making research more available, transparent, & traceable. We encourage the sharing of data (where ever possible) & analysis scripts, and the introduction of preregistration, registered reports, large scale collaborations, the TOP guidelines, etc.

>We explicitly criticize QRPs such as p-hacking, harking, story telling, or the general incentive structures that prioritize publication quantity over research quality.

>Although we explicitly focus on quantitative approaches in the hypothetico-deductive context, we also offer some tentative suggestions how #opencomm might work for the more qualitative approaches.

(for more information, see talk by @lindseybier youtube.com/watch?v=nm-198…)

>So, again, I was VERY curious re. how the agenda would be received by our community.

Overall, the feedback was overwhelmingly positive, and I'm really happy & thankful for all the nice & kind words we've received :)

>However, both in our reviews from JOC and the subsequent discussion at #ica20, we did get a bit of pushback coming from one particular direction -- which, tbh, somewhat surprised me.

>(But, to be crystal clear here: Feedback was fair, constructive, good spirited, and self-reflective -- just the way it should be.)

>Interestingly, the agenda was not criticized by our field's industrious p-hackers, which was what happened in psych, and for whom consequences indeed are dire (mb they've learned from the mistakes of their psych kin, bc arguments simply don't hold up?)

>Instead, pushback came from the more qualitative approaches. I've read the following objections & concerns:

(pls. feel free to correct wrong understandings or to add novel points.)

>1. Replicability not relevant, bc. all knowledge is contextual. 2. Through self-reflectively explicating lenses & biases, qual. is already doing most of this anyway. 3. The researcher is part of the knowledge process, hence cannot be separated from the results.

>4. Preregistration doesn't make sense, bc. we don't know what we'll find beforehand. 5. Knowledge is co-created together with participants, hence it's a process that cannot be planned/explicated/replicated. 6. Results need to be interpreted and are never absolute/general.

> 7. It's not about replicability or generalizability, but transferability. 7. In a lot / most (?) cases, sharing data is not possible/useful: Too risky for participants, even when anonymized/deidentified, and too much effort for too little value of what's left.

>Here's my reply:

>Qualitative research is not the *problem*. Qualitative research is the *solution*.

>Exploratory research is also not the problem. The problem is selling exploratory research as confirmatory. Exploratory research is CRUCIAL for knowledge building. Exploratory research is part of the *solution*.

>Preregistration & Registered Reports do *not* prevent exploration. This is a common & false misunderstanding. Instead, by explicitly labeling unplanned/emergent/novel analyses as exploratory, we make them *more* prominent. We elevate them.

>Preregistration is mainly for hypothetico-deductive, quantitative approaches. If you're 100% exploratory, just skip preregistration! It's *not* mandatory.

But most of the time we start with a theoretical framework, perspective, assumption, or hunch -- this you can preregister.

>Yes, you absolutely cannot share specific kinds of data. End of story.

>Yes, qualitative research is *much* more contextual. BUT: It's not a 100% contextual. Bc if it were a 100% contextual/idiosyncratic, we could, by definition, not transfer it. But we can.

>And, granted, we do: Just read the discussion sections, you'll often find many (relevant!, important!) claims re. underlying power structures, generally disadvantaged groups, oppression mechanisms, etc.

>BUT: You cannot have the cake and eat it. It's a truism that for the very same reasons, the transferability of insights coming from small scale qualitative studies is also more limited.

>Let's not forget: Objectivity >= Reliability >= Validity.

Let's also not engage in jingle-jangle fallacies: Transferability ~ generalizability.

>Awareness of reduced generalizability is not a problem, it's again part of the solution. Quant researchers can learn much from qual researchers:

Our knowledge is highly contextual. Don't overclaim. Know yourself. Know the structure you're working in. Be humble.

(The irony.)

>I'm perfectly aware that all this is highly delicate: E.g., there are different understandings of what qual. research itself even is. In our agenda, we explicate that. I'm no expert on qual. I'm hence happy to have other jump in here. These are my thoughts, my understanding.

>I honestly believe that the replication crisis has shown that we should be more humble, more descriptive, more nuanced. If I'm not mistaken, these are core values of qual research.

>I would love to encourage qual research to see the many opportunities #openscience offers. This is really not about paternalizing qual. & exploratory research. This is about *empowering* qual & exploratory research!

>Don't believe me? Well, then read @talyarkoni magnificient paper "the generalizability crisis".

(Background: Tal is a leading figure of the #openscience movement in psych and, without doubt, currently the smartest person living on this earth).

This is what he writes:

>Conclusion: #openscience and #opencomm will lead to knowledge claims that are more robust. In this process, qualitative research is not part of the problem but part of the solution.

>Let's all join this discussion, continue to exchange thoughts and concerns, see what works and what not. I truly believe that #openscience has much to offer for *all* of us.

> (Please don't kill me.)

(To read this on one single page, see tobiasdienlin.com/2020/05/24/ope…)