,
9 tweets,
4 min read
Read on Twitter
Measurement validity is bonkers!🤨 Writing a thread here in response. Thnx @carole_lunny for the prompt! Starting w/ definition of validity. Need to consider validity as interpretations of test score use (ie.validity as "measuring what it is supposed to" has gaps in meaning). 1/9
2/ Article by @EikoFried and @JkayFlake is excellent...e.g. hidden invalidity is so on point. Text by Devellis (2017) is very good for scale development BUT ignore chapter on validity. ***Note: many texts use unreferenced validity theories and out-of-date notions of validity.
3/Great chapter by Bandalos (Ch.11) in text Meas. Theory in Social Sciences! Has a nice table comparing traditional to current views. Careful bc much evidence only considers validity as correlations - this is fine when relating test to other measures but alone not enough.
4/Ask yourself this when you read info, “how does validity info enhance our understanding of the <scale>”? (Quote from me in section about strengthening evidence as we evaluate validation practices in this paper w/@sheilakmarshall @BD_Zumbo ): journals.sagepub.com/doi/pdf/10.117….
5/Many disciplines (& even many texts) have not embraced current views and also have no rationale or reference for their validity approach. Imo, this is problematic as gathering validity evidence has become kinda ritualistic and needs more critical thought & discussion.
6/For some theory & to see evolution to current (unified) view go back to Cronbach & Meehl (1955). Loevinger (1957), Messick (1989, 1995), @BD_Zumbo (1989, 2017) & Kane (1990, 2006) are some of the big players. See also Test Standards (AERA, APA & NCME, 2014).
7/Traditional views are fine but often no justification for use or ref. to theory. Eg. I have yet to see a good argument why a latent (ie.unobservable) construct has a criterion (“gold standard”).Criterion for a behaviour makes sense but not for a construct like self-efficacy 🤔
8/My fav way to talk about validity is to build a validity argument (credit to Kane for analogy) & whether evidence is strong/weak for test use. Gathering validity as a checklist often talks of “types”, which gives little info on inferences/interpretations we make about scores.
9/Validity is complicated and so much to sift through - definitely lots to chew on & takes time to get a good handle on and digest. But so important esp. when it comes to good science. Also relevant for embodying patient- or student-centered practices. Hope this can help! 🤓 -end
