Thread by @Kunkakom

In a new study at #APSR, we show how established operationalization strategies of multi-dimensional concepts can systematically lead to wrong conclusions.
Focusing on populist attitudes, we demonstrate simple methods to align theory and measurement.
👇Summary+Preprint+Shiny App

Our (H. Schoen+@ChristianSchim) argument refers to a specific but common type of multi-dimensional concepts which are sometimes called ‘non-compensatory’. Multi-dimensional concepts are non-compensatory when higher values on one component cannot offset lower values on another.

Think of democracy: If we believe that a country only counts as a democracy if it provides both rule of law and free elections then no valid measure of democracy will assign high democracy scores to a country with low 'rule of law' scores even with the election are very,very fair

Not all social science studies consider this seemingly minor but consequential issue when putting multi-dimensional concepts into empirical practice. Focusing on populist attitudes, we show that these measurement-concept inconsistencies can lead to wrong conclusions.

Most importantly, we demonstrate easy-to-implement operationalization strategies that account for the non-compensatory nature of (some) multi-dimensional concepts, for both binary and continuous concepts.

We tried hard to transparently report all evidence for our claims and to give as much guidance as possible in case you want to apply our proposals in empirical practice.
Main study+Supplement=122 pages😬
Also, we have a shiny Shiny Web Application so you can play with the data🤗

Let me run you through the argument in more detail. We focus on populist attitudes (PA) but the methodological arguments also apply to other multi-dimensional concepts.
Preprint of the Study: alexander-wuttke.de/wp-content/upl…
Shiny Application: populism.alexander-wuttke.de

Populism is an essentially contested concept. Yet, most scholars now agree that PA are multi-dimensional (e.g.: anti-elitism + Manicheanism + Sovereignty). Importantly, there is also widespread agreement on the idea that PA lie at the intersection of the concept’s sub-dimensions.

Hence, the unique property of PA is the co-existence of its components. Put differently, we only consider citizens (or leaders) as populists if they accept anti-elitist views AND a Manichean outlook AND support popular sovereignty.

Importantly, it is this non-compensatory concept property that distinguishes PA from other established public opinion constructs (eg cynicism, efficacy, ethnocentrism), that makes PA worthwhile as a concept and that makes it more than the sum of its parts.

Yet, existing studies on populism at the mass level rarely transfer this crucial concept feature into empirical practice. Hence, some populism studies do not measure what they intend to measure and reported results do not necessarily speak about the concept under investigation.

We argue that the most-often used operationalization approach (CFA or average scores) is rooted in a measurement paradigm that is often applied to latent constructs and which implicitly views the relationship between concept and concept components as causal (‘Bollen approach’).

We argue that a different perspective is needed when multi-dimensional concepts are non-compensatory. This perspective allows for operationalization strategies that account for this concept property (we call them the Sartori and the Goertz approaches)

A straightforward Goertz-procedure is to use the minimum value of the concept components. The Sartori-approach entails setting thresholds on each concept components. Both approaches ensure to only assign high values to individuals if they score high on ALL concept components.

These distinctions may seem like nitpicking. But they can make a crucial difference for substantive conclusions concerning nature and correlates of populist attitudes.

Our preferred operationalization approach (Goertz) and the established approach (Bollen) result in different populism scores.
Correlations between them are 0.4 to 0.9
Note: these scores were derived from the same data-generating process. They only differ in the aggregation rule!

Consider institutional trust.
Bollen composite scores suggest that higher levels of populist attitudes go along with lower levels of trust.

Yet, apparently, this association is driven entirely by anti-elitist orientations and not by the distinct concept of PA as an attitudinal syndrome at the intersection of ALL subdimensions.

When operationalizing PA in a way that accounts for the non-compensatory relationship of the subdimensions (Goertz), then the seeming association between institutional trust and populist attitudes disappears (see plot above).

We examined a large number of correlations in many datasets with different populism scales.
In most cases, Goertz index indicates weaker associations with substantive variables than Bollen.
In many cases, the conclusions do not differ.
But too often they do,sometimes drastically.

Quick detour to our shiny Shiny App:
You can examine yourself how the disparities of the Sartori, Bollen and Goertz concept structures vary with scales and countries.
Also, select a country and scale of your choice and see correlations with variables of interest.

In addition, we use the Shiny App to demonstrate the relevance of researcher discretion. Specifically, using the Sartori approach we show how the estimated share of ‘populists’ various with specific details of the operationalization.

Have a look at the Shiny Web Application!
An unknown share of users cannot open the website because of university firewalls. Please let us know when you receive an error message and tell us if you are using the university network.
populism.alexander-wuttke.de

If your research deals with (non-compensatory) multi-dimensional concepts you might want to have a look at our step-by-step guide in Supplement 2.

If you study populist attitudes (using the Schulz et al. Akkerman et al., Castanho Silva et al., Oliver/Rahn scale or the CSES scale of populist attitudes) you might want to have a look at Supplement 6 in which we discuss each scale and suitable operationalization strategies.

The study is scheduled to be published Open Access in January.
What a ride. This was a truly fabulous collaboration with H. Schoen + @ChristianSchim.
Let us know if you have any questions or if we made any mistakes.

@ChristianSchim Let us add that various twitter-people deserve our gratitude
@denis_cohen helped us with the simulation stuff that was above our head
@b_castanho et al made the analysis possible by sharing their data
And many others (eg @Flavio_Azevedo_,@uzh_ikmz, @spasspolitik) for feedback