1/n This project is finally updated! We argue that generative modeling can improve the quality of inference on behavioral data, and we use simulations and empirical data from the Stroop, Flanker, Posner Cueing, Delay Discounting Task, and IAT to show it.
2/n Recently, many papers have been published showing that traditional analyses lead to poor reliability for behavioral measures. We argue that there are both theoretical and statistical issues with traditional methods (e.g., mean contrasts) that are improved by generative models
3/n We then give an example of how background knowledge can be used to inform generative models of person-specific behavior. Our example uses lognormal and shifted lognormal models, the parameters of which make specific predictions about the shape of a person's response times:
4/n The goal of generative modeling is then to fit these generative models to individual-level behavior and use the resulting parameters (i.e. difficulty and dispersion) to make inference on mechanisms underlying behavior. But that is not all!
5/n By fitting the generative models to all participants simultaneously in a hierarchical fashion, we can account for measurement error at the individual level that would otherwise produce biased inference and low test-retest reliability. Our supplemental simulations show this:
6/n What does this mean for empirical data? Well, when we use the generative models to infer test-retest reliability across a variety of behavioral tasks, generative models improve reliability consistently in every example (here is the Stroop task):
7/n and the Flanker task (black lines in the left panels indicate the traditional mean/SD contrast reliability):
8/ And the Posner Cueing task:
9/n And the Implicit Association Test (Black/White Race version):
10/n And the Delay Discounting task (which shows that generative modeling is useful outside of response time tasks, given that we only modeled preferential choices in this example):
11/n We also compare the person-specific estimates derived from the traditional two-stage summary statistic approach to the generative models to see how the hierarchical model induces regression toward the group-level mean, in turn producing more precise person-level estimates:
12/n We end with some recommended resources for others who would like to pursue generative modeling in their own research:
13/n And with a cautionary note:
14/n I hope that these results help convince others that we can learn a lot from behavior! But it requires a shift away from atheoretical summary stats, toward generative thinking. Thanks to @PeterKvam, @colinsmithpsych, @wooyoungahn, and others for putting up with me 😊🤗
15/n All the data, R, and Stan codes are available on GitHub if you are interested in checking them out: github.com/Nathaniel-Hain…
dangit I always forget that github posts the awkward profile pic.. oh well lol
I should have added here one of my favorite paragraphs—summary stats can lead us astray even when they produce highly replicable results:
I should have also noted that there are no data-preprocessing tricks used to get these results. We used all trials for every person in each task (except for a few [like 8 in total] negative RT trials in the Posner task). Modeling all the data is key from a generative perspective.
May be of interest to #measurementschmeasurement researchers, @EikoFried, @JkayFlake, @KMKing_Psych, @Sam_D_Parsons 🤖
And also those doing idiographic work given the focus on individual differences, @aaronjfisher, @JulianBurger, @aidangcw, @MaddyFrumkin
