Thread by @ingorohlfing

There is a highly instructive symposium on "Veil of Ignorance" #ProcessTracing in the new @APSAtweets QMMR publication qmmrpublication.com/uploads/1/1/4/… There is lots to say ab it. For now, I want to focus on whether (causal) qualitative research has a problem w/ confirmation bias that 1/

needs to be addressed.
In short, Veil of Ignorance process tracing aims to blindfold the data collectors and analysts to the hypotheses to reduce risk of confirmation bias.
Proposal receives a lot of criticism by all respondents, partly for valid reasons, partly not, IMO 2/

If there was no bigger issue with confirmation bias, there would not be much benefit in talking about blinding in qualitative research.
As Beach notes, the intro to the symposium does not present evidence that bias is a problem. This is correct, but there is also no evidence 3/

that confirmation bias is not a problem. It's an absence of evidence situation with regard to comprehensive, direct evidence, i.e. from verification studies of qualitative research (for studies that are built on an epistemology for which the idea of verification makes sense). 4/

However, I'd say there is circumstantial evidence that confirmation bias is a problem.
1) Evidence for confirmation bias in quantitative research: The developments around #OpenScience and transparency in quantitative research in different disciplines present diverse evidence 5/

for confirmation bias (aka as publication bias).
- Selective reporting/underreporting: doi.org/10.1093/pan/mp…
- Unusual excess of positive results: doi.org/10.1177%2F0049…
Could it be different for qualitative research? Unlikely. Publication bias, whatever its source, 6/

creates an incentive for positive results because careers, funding etc. depend on getting published. I don't see why the same incentives shouldn't apply to qualitative researchers (more 👇). I neither find it plausible that researchers who are less susceptible to these 7/

incentives select themselves into the application of qualitative methods.
2) Publication bias in qualitative articles
@alan_jacobs1 has published a terrific book section on transparency in causal qualitative research politics.sites.olt.ubc.ca/files/2018/11/… It presents clear evidence for 8/

publication bias in qualitative research. This is not direct evidence for confirmation bias in these articles bc it could also be that articles w/ negative results are filtered by peer review. However, it is a sign that confirmation bias could be an issue. 9/

3) A partial reassessment of Moravcsik's 'A Choice for Europe'
Lieshout/Segers/van der Vleuten presents a reassessment of a specific part of Moravcsik's monograph 'A Choice for Europe'
jstor.org/stable/26925425
I find this article is an excellent illustration of how one can 10/

verify qualitative research. Their reassessment points to a couple of problems with the misinterpretation of sources being the biggest one. This leads them to conclude that Moravcsik's original conclusion about some of de Gaulle's policies does not hold closer scrutiny 11/

Overall, it is true that there is no good evidence about confirmation bias in qualitative research. However, my reading of the evidence that we have is: We should be concerned about bias because different pieces point to its presence. 12/

This does not mean that Veil of Ignorance process tracing is necessarily a good technique for addressing it. However, it points to the broader importance of reflecting upon potential ways to detect and diminish bias in qualitative research (maybe more on this [much] later) /end

There is one more: A big 👏to Jennifer Cyr @policentrica for having put together this symposium and running the QMMR publication

@threadreaderapp unroll