,
15 tweets,
4 min read
Read on Twitter
I think that this new paper is a great example of the type of meta-scientific work that we need to evaluate the effect of changing research practices. But I worry about the simple conclusion that "the push for better practices makes us use cheaper and easier methods" 1/15
We've done similar coding; haven't published it yet, because each time we get into it, we realize that this type of coding is hard, and we keep wanting to make different decisions. So my conclusions are based on coding that I now think is not ideal, but I'll continue anyway 2/15
First, I think this conclusion is problematic. This is a causal conclusion that the push for bigger Ns has led people to rely on Mturk and other on-line studies. The authors do acknowledge later that there are other possibilities; but this is the headline that will stick 3/15 
Yes, online studies have increased...but a big reason for that is that Mturk pretty much only became available right as concerns about replicability emerged. Pretty clear in this plot 4/15
At the same time (and this is not included in their paper), reliance on another low-cost, relatively easy-to-obtain population has decreased: studies rely on undergrad samples to a lesser extent now than they used to 5/15
And, overall, if we just look at the percent of studies relying on either one of these relatively-easy-to-obtain samples, then things haven't changed too much. 6/15
What about reliance on self-report and the shift away from behavior? Well here's where we screwed up (we didn't code *exclusively self-report* or certain types of behavior). But this is also where we need much more detail about what old and new studies actually did. 7/15
Because in our analyses, even early studies were pretty bad at doing anything other than self-report + some button-pushing. Few studies with longitudinal methods, informant reports, physiological measures, dyadic designs, experience sampling, even before push for large N. 8/15
So I'd love to see these authors (or someone) go into exactly what types of *difficult/expensive* methods were being used and whether those have actually declined over time. 9/15
The message people may hear is that "we need to go back to the good old days where we measured actual behavior" but I'm not so sure the good old days were all that good. 10/15
Couple other things to point out. First, I noticed this quote in the methods. But this is part of the problem. A good way to address concerns about small N is to think about novel ways to test hypotheses, including using high quality data that someone else collected 11/15
In fact, one pattern is that JRP didn't rely on Mturk samples & was a little better than JPSP in reliance on undergrads. Ns at JRP also increased during this time & were already larger to begin with. So increases in N can happen without corresponding reliance on internet 12/15
Why? Probably because personality psychologists dont rely solely on experiments where researcher has control over every factor. They can use panel studies like the GSOEP or HILDA to look at how variables that other researchers decided to include are linked to one another 13/15
But so many social psych phenomena could also be informed by nonexperimental methods and the data sources that personality psychology (and many other social sciences) use. So another solution to the problem of larger N: Move beyond traditional way of doing social psych 14/15
Finally, as many others have noted, if we need big Ns but single labs don't have resources to use difficult methods, the solution is not to settle for one or the other. It is to develop novel ways of getting these studies done, like @PsySciAcc and @Study_Swap are doing 15/15
