,
7 tweets,
3 min read
Read on Twitter
1/7 In 2017, Hedge, Powell, and Sumner showed that robust cognitive tasks are unreliable, which calls into question the use of behavioral tasks for studying individual differences. In this blog post, I show that this conclusion is misguided (haines-lab.com/post/thinking-…) 
2/7 Specifically, Hedge et al. found that robust effects such as the Stroop effect have test-retest correlations in the range of .5 to .6 (r = .5 in the plot of their Stroop effects shown here), which severely impacts our ability to rank the performance of individual subjects.
3/7 However, this conclusion is based on a model of behavior does not account for variability at the individual-subject level. While this may not seem very problematic at first glance, the assumption of no measurement error leads to substantially biased inference.
4/7 In fact, a hierarchical model that simultaneously pools information across subjects while estimating reliability results in test-retest estimates as high as r = .95 (posterior distribution of test-retest corr shown here, along with the pooled versus raw sample means).
5/7 Further, using raw sample means from indiv. subjects in a secondary analysis is equivalent to artificially replicating your dataset ad infinitum before fitting your model (fig showing pooled effects converging to sample means with increasing replications).
6/7 In conclusion, averaging your data within subjects before modeling the resulting averages is as functionally problematic as duplicating your dataset an arbitrarily high number of times before fitting a model that appropriately accounts for within-person variability!
7/7 All #rstats and #stan codes included in the post so that you can easily replicate. If you are interested in more of these ideas, I highly recommend checking out Rouder & Haaf (2019) (link.springer.com/article/10.375…), which I unfortunately only discovered after finishing this project!
