So, there are problems with the analysis by @epkaufm & @Goodwinmj (K&G) in their Quillette defense of their now infamously titled @unherd event on immigration: 1/15
K&G motivate their analysis by a desire to bring evidence to a highly polarised debate and to subject the claims made by their critics to ‘measurement and data’ as opposed to ‘confirmation bias and motivated reasoning’. Hmm. 2/15
First issue up is that they effectively ignore their main experimental result – people *did* find the event title offensive. The difference between their treatment and control conditions is large and statistically significant as you can see here 3/15 
But K&G glide over this to focus instead on showing that there was no significant difference in offence between whites and non-whites within their Diversity Treat treatment group 4/15
But there are at least 3 problems with this 5/15
Problem 1 is that, whether there is a difference between ethnic groups in level of offence is not the appropriate test of whether people, including ethnic minorities, found the event offensive. They did. 6/15
Problem 2 is that the relevant experimental quantity here is not the difference in offence between ethnic groups within the treatment group but the *difference in the difference* between treatment & control groups. K&G don’t consider 7/15
Problem 3 is, even using the treatment group as an observational study, as K&G do, the study design is woefully underpowered to detect a difference between whites and non-whites; their sample of 41 whites and 38 non-whites is just too small 8/15
To illustrate: for a 5 point difference between whites and non-whites, K&G’s design has power of 0.1 and, for a 10 point difference, power of 0.3. Even a difference of 15 points has power of only 0.58. The recommended minimum power for clinical trials is 0.8 9/15
The exact same problems pertain for the claims K&G make relating to the effect of political ideology on offence. They don’t make an experimental comparison but, instead, use their treatment group as if it were a general population sample. But it isn’t. 10/15
It’s a *really weird* unweighted sample that mixes UK & US respondents in an unexplained way and which has some v off-kilter distributions. What is the target population here? I’ve no idea 11/15
And again, the sample size is way too small to say anything meaningful about between group differences. The ‘extremely liberal’ group that K&G focus on? That’s based on just 8 respondents. Yes, that's 8 respondents 12/15
And K&G’s sample is so small they don’t even observe a single ‘extremely conservative’ person at all, though presumably such people exist in the population (whatever that population is – see 11/15 above)? 13/15
The key point is, if you want to base your conclusions on failing to reject the null hypothesis, you need to ensure you have sufficient power to do so. K&G don't 14/15
And, if you rebuke others for making unsubstantiated & ideologically motivated claims, it’s as well to ensure that your own conclusions are justified by your analysis and data. Which is ironic when you consider this excerpt from K&G's Quillette piece /end.
p.s. credit to K&G for making data available and answering my questions promptly.
pps I haven’t said anything about treating ‘non-whites’ as a single ethnic group but that’s clearly a bit of a weird thing to do
