Thread by @Kunkakom

What can you do when an experiment yields null results?

I developed a theory to explain why (some) people derive pleasure from political engagement
14 out of 15 predictions failed

▶️A step-wise process to maximize the informational value of null results
doi.org/10.1017/pls.20…

The need-based theory of political motivation in a nutshell:

Building on the ‘Pleasure Principle’, I suspect that people will seek behaviors that made them feel good in the past.
Citizens will therefore seek political engagement when it previously made them feel good.

To understand what actions provide pleasure, I built on the concept of basic psychological needs.
Differences in intrinsic political motivation are theorized to reflect whether political engagement was previously experienced as satisfying basic psychological needs.

Popper taught us that we cannot confirm scientific theories.
After having conducted a pre-registered, well-powered experiment I now know what other philosophers in response to Popper had already said decades ago:
Falsifying a theory is also hard to impossible.

14 out of 15 empirical results were not consistent with my theoretical predictions.
But does this prove the theory is wrong? No, because so many explanations could account for the hypotheses-inconsistent findings.

In contrast to the usual strategy of strengthening a finding to avoid a false-positive, the goal in the case of null results is to avoid a false negative.
Hence, I conducted various posthoc analyses to test rivaling explanations for the absence of an observed effect.

▶️Analyses on reliability and treatment-induced attrition suggested that measurement issues likely have not hidden true effects
▶️Also, I used machine-learning approaches by
@Susan_Athey et al. to assess whether effects might be hidden in subgroups

▶️I used equivalence tests by @lakens, @annemscheel,
@peder_isager to ensure not to miss any potentially meaningful but insignificant effects
▶️Finally, I checked various design features, showing that indeed only one experimental arm can be understood as a valid test

In hindsight, I would now likely also add Bayesian analyses to the list of measures for reacting to or preparing for null results. We are learning more and more about how to make null findings informative. That's great.

All of these tests increased my confidence in the conclusion that, indeed, there is something wrong with the theory itself in its current form. Hence, it likely requires refutation or revision.

Unfortunately, the study is therefore not the big leap for research on intrinsic political motivation that I once hoped it would be.
I discuss what we still can learn for how to study why some people find pleasure in politics and others do not. /END