Thread by @uygun_tunc

In the debate on "social” priming in Psych Inq, it was maintained in the target article (by Sherman& @amrivers1 ) and several commentaries (e.g. @missyjferguson & @JeremyCone2; @dalbarra &Wenhao Dai ) that we must investigate “operating characteristics” and “moderators”.🧵1/20

The problem of moderators we keep facing here as well as in many other literatures signals in fact a deeper and prevalent problem to which many authors have for decades drawn our attention: Except a few instances, psychologists almost never test their auxiliary hypotheses. 2/20

Auxiliary hypotheses are those we, out of theoretical and methodological necessities, assume to be true as we test our main hypothesis

https://twitter.com/uygun_tunc/status/1342154375893020673

. To exemplify, the target article mentions the auxiliary hypotheses > 3/20

that the length of delay between prime and target does(/not) influence the effect of priming, dichotomous, multiple choice and continuous measures of behavior as DVs are(/not) equivalent, or that a within subject design is(/not) equivalent to a between subject design > 4/20

(For instance Fritz Strack & @aanobs indicate that within subject design has the risk of providing cues for the concerning the purpose of the study: tandfonline.com/doi/pdf/10.108…). 5/20

When these and other auxiliary hypotheses aren't sufficiently tested beforehand to illuminate how they will influence the results, we can hardly interpret what the results imply for the main hypothesis, be they (apparently) consistent or inconsistent with the main hypothesis.6/20

Because in both cases the results can be blamed on auxiliary hypotheses. Thus, falsification or corroboration of the main hypothesis becomes nearly impossible. This problem is in fact a familiar one in phil sci; namely, empirical underdetermination. 7/20

https://twitter.com/uygun_tunc/status/1260651503909363713

The problem of underdetermination proves to be particularly difficult in the social sciences, where auxiliary hypotheses are harder to test independently, and paradigmatic theories don’t organize the fields in the form of strong nomological networks. 8/20

The social sciences isn’t oblivious to this problem either. Meehl emphasized its importance repeatedly and in strong terms (doi.org/10.2466/pr0.19…) If he lived a little longer, he would also work on his big plan to catalogue all important auxiliary hypotheses in psychology. 9/20

More recently several other authors made important remarks on how the common practices of hypothesis testing in psychology need to be reformed with emphasis on the role of auxiliaries: 10/20

@briandavidearp & Trafimow (2015) and Trafimow (2012) address the ambiguity of failed replications and falsifications of theories due to inadequate attention to auxiliary assumptions in designing replications and theory tests 11/20 frontiersin.org/articles/10.33…& doi.org/10.1177/095935…

@annemscheel et al (2020) explicate how premature testing of hypotheses lead to the accumulation of uninformative results 12/20 pubmed.ncbi.nlm.nih.gov/33326363/

@gershbrain (2019) addresses how ad hoc auxiliary hypotheses protect beliefs from disconfirmation 13/20 link.springer.com/article/10.375…

What has changed after all these calls to place more emphasis on the role of auxiliary hypotheses? Nothing much. There is still little awareness of the need for explicitly specifying auxiliary hypotheses in designing strong tests 14/20 frontiersin.org/articles/10.33…

Since nothing has changed, we have to have the same debates over and over, just like the debate on “moderators” in the “social” priming literature. Psychology can and should do better. 15/20

https://twitter.com/lakens/status/1388021436724428803

Not to sound alarming but psychology might long have been facing an auxiliary hypotheses crisis. The first thing to do is to take the problem seriously, and to understand that it is not solved by saying “it would also be good to think about the potential moderators”. 16/20

Psychology’s chronic problem of underdetermination won’t be solved unless auxiliary hypotheses are examined in an organized and systematic manner. Otherwise there will be special issues on the same research questions every 7 years. This is indicative of huge research waste. 17/20

With half of the effort and resources that go into creating this research waste, psychology can build much more reliable and credible literatures. As a way to tackle this problem, I and @tunc_necip humbly suggest the systematic replication framework (SRF): psyarxiv.com/pdm7y/

SRF is a (sophisticated falsificationist) method to reduce the ambiguity of the results of hypothesis tests by eliminating alternative explanations findings due to auxiliary hypotheses. Main points and summary can be found in this thread:

https://twitter.com/uygun_tunc/status/1342154373456130048

19/20

SRF or alternative systematic approaches might at first sound too costly and impractical. But then just think about the costs and feasibility of the current methodological status quo in terms of the research waste it keeps on creating. 20/20