Thread by @JkayFlake

I just read this paper: psyarxiv.com/7rbfp/ by @ianhussey + Sean Hughes, "Hidden Invalidity among Fifteen Commonly Used Measures in Social and Personality Psychology" - Let me tell you why it is important.
THREAD 1/n

2/n first, this is a massive study of the psychometric properties of 15 scales that are used in social and personality psychology. If you want to use one of these scales you have a comprehensive evaluation of critical psychometric information right here

3/n given there are many scales out there, in use, without such evaluation, this makes a big contribution to the substantive areas that use these scales, but even if you aren't interested in these scales, there are some key takeaways for the whole field to consider

4/n a critical one is that reliability coefficients seem to be used to "hide" validity issues with scales. Despite that scales meet common cutoffs for alpha, they do not show strong psychometric properties needed to even use alpha (are not unidimensional)

5/n reliability coefficients don't tell us anything about if a scale score represents the construct we set out to measure, quit it with this; when you review, demand more information! If you didn't realize this, change your ways.

6/n some of these scales failed tests for configural invariance across gender-this means that the basic configuration of items mapping to factors was different across genders, --->

7/N THIS MEANS THAT MEN AND WOMAN ARE INTERPRETING THE ITEMS SO DIFFERENTLY THAT THE CONSTRUCTS MIGHT NOT EVEN HAVE THE SAME MEANING ACROSS GROUPS. We can NOT adjust for this with SEM, we have to deal with the theoretical implications of it

8/n if you read this carefully it will really getting you thinking about how we can cherry-pick psychometric information to support our claims that scales are valid, and even if that information is scant, we get away with it. Pre-reg, reg reports, it is for scale work too

9/n There is a lot of work to be done in developing transparent and thorough practices that support the valid use of measures, we are just getting started here. It isn't easy, there is statistical and theoretical complexity, rules of thumb won't save us

10/n My short term advice is, think deeply about what you study and how you measure it. When reviewing, ask for more information, in your own research ask yourself hard questions, if you don't know the answers, that is okay, this is science, just be transparent about it.