Thread by @RubinPsyc

New article:

“Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests.”

Published #openaccess in Philosophical Psychology @JournalPHP: doi.org/10.1080/095150…

#openscience #philsci #metascience

🧵👉

Researchers often distinguish between:

1⃣ Exploratory hypothesis tests - unplanned tests of post hoc hypotheses that may be based on the current results, and

2⃣ Confirmatory hypothesis tests - planned tests of a priori hypotheses that are independent from the current results

This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results.

We challenge this assumption and argue that exploratory results can be more compelling than confirmatory results.

In the first part of our article, we demonstrate that the same data can be used to generate and test a hypothesis in a transparently valid manner.

We agree that circular reasoning can invalidate some exploratory hypothesis tests. However, circular reasoning can be identified by checking the *contents* of the reasoning without knowing the *timing* of that reasoning (i.e., a priori or post hoc).

In the second part of our article, we argue that exploratory hypothesis tests can have several evidential *advantages* over confirmatory tests and, consequently, they have the potential to deliver more compelling research conclusions. In particular,...

Exploratory hypothesis tests:
✅ avoid researcher commitment and prophecy biases
✅ reduce motive for data fraud
✅ are more appropriate following unplanned deviations
✅ facilitate inference to the best explanation
✅ allow peer reviewers to contribute to exploratory analyses

Finally, in the third part of our article, we consider several potential *disadvantages* of exploratory hypothesis tests and conclude that these potential disadvantages may not be problematic. In particular,…

Exploratory hypotheses tests are not necessarily disadvantaged due to:

✅overfitting
✅bias
✅HARKing
✅unacceptable research practices

And they:
✅are usually necessary
✅can be falsified
✅can predict anything but may suffer an evaluative cost in doing so

To be clear, our claim is not that exploratory hypothesis tests are *always* more compelling than confirmatory tests or even that they are *typically* more compelling. Our claim is only that exploratory tests *can be* more compelling in specific research situations.

More generally, we encourage researchers to evaluate specific tests and results on a case-by-case basis rather than to follow simplistic heuristics such as “exploratory results are more tentative,” which represents a form of methodolatory:

https://twitter.com/RubinPsyc/status/1532956108155142144

Our paper builds on some of my previous work on preregistration and HARKing.

doi.org/10.20982/tqmp.…

drive.google.com/file/d/1bGIUjH…

For my other articles in this area, please see: sites.google.com/site/markrubin…

And please check out Szollosi and Donkin’s (2021) paper on “the misguided distinction between exploratory and confirmatory research.”

doi.org/10.1177/174569…

Finally, we’d like to thank our peer reviewers and the editorial team at Philosophical Psychology @JournalPHP for providing an efficient and constructive review process that greatly helped to improve the quality of our article. 🙏

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology. doi.org/10.1080/095150…