New article:

“Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests.”

Published #openaccess in Philosophical Psychology @JournalPHP: doi.org/10.1080/095150…

#openscience #philsci #metascience

🧵👉 Researchers often distinguish between:

1⃣ Exploratory hypothesis tests - unplanned tests of post hoc hypotheses that may be based on the current results, and

2⃣ Confirmatory hypothesis tests - planned tests of a priori hypotheses that are independent from the current results

Here’s a (non-exhaustive) list of 10 (potentially compatible?) explanations for the replication crisis.

THREAD >>> (1) Low statistical standards of evidence

“Statistical standards of evidence for claiming new discoveries in many fields of science are simply too low” (Benjamin et al., 2018).
doi.org/10.1038/s41562…

Some potentially useful recent papers on stats and methodology... André, Q. (2021). Outlier exclusion procedures must be blind to the researcher’s hypothesis. Journal of Experimental Psychology: General. doi.org/10.1037/xge000…

My new paper considers: “When to Adjust Alpha During Multiple Testing”

I consider three types of multiple testing and argue that an alpha adjustment is only required for one of them.

Pub: doi.org/10.1007/s11229…

Free: rdcu.be/cnSpI

OA: drive.google.com/file/d/1Rl23xQ… I argue that an alpha adjustment is *not* necessary when undertaking a single test of an individual hypothesis, even when many such tests are conducted within the same study.

Preregistration helps to distinguish planned "confirmatory" tests of a priori hypotheses from unplanned "exploratory" tests of post hoc hypotheses. However, some people argue that this distinction doesn't really matter.

Let’s talk about that!

A THREAD… Some people argue that the *type* of hypothesis generation (deductive vs. inductive) is more important than the *timing* of hypothesis generation (a priori vs post hoc; e.g., Worrall, 1985, 2010, 2014).

***HARKing***

A THREAD

HARKing stands for Hypothesizing After the Results are Known. It occurs when researchers present their post hoc hypotheses as if they are a priori hypotheses. Kerr (1998) wrote the seminal article on HARKing…
doi.org/10.1207/s15327… HARKing has been described as one of the four horsemen of the replication apocalypse (Bishop, 2019). But is it really that bad?

There’s been some nice work recently that has taken a critical look at the theory and practice of preregistration. Here’s what I think are some key articles. (Plus my own wee contribution!)

(Thread, 1/9) Devezer et al. (2020). The case for formal methodology in scientific reform.

doi.org/10.1101%2F2020…

(2/9)

My new paper - Explaining the association between subjective social status and mental health among university students using an impact ratings approach.

Publisher’s full-text view only: rdcu.be/cbWOn

Self-archived version: drive.google.com/file/d/1fodJBc… Prior research has found a positive association between social class and mental health among uni students. Various mediators of have been proposed. But the extent to which students perceive these mediators as having an impact on their mental health has not been looked at.