Thread by @RebeccaSear

Do 'national IQ' datasets present accurate and unbiased data on average IQs in nation-states worldwide?

A thread 👇

(Spoiler alert: NO)

1/n

A recent paper, published in a peer-reviewed journal, which used such a dataset prompted me to examine the most recent version of the Lynn & Vanhanen 'national IQ' dataset, which purports to estimate average IQ for a large proportion of countries worldwide 2/n

In 2007 I co-authored a critique of a paper which used an earlier, 2002, version of this dataset, which involved examining the dataset in detail. Our conclusion: “The primary data are grossly inadequate...the sampling is sketchy at best and ludicrously insufficient at worst” 3/n

We looked particularly at the samples used to estimate the 'national IQs' of 3 countries which had exceptionally low estimates: Ethiopia, Nigeria and Sierra Leone. What we found wasn't good 4/n rebeccasear.org.uk/wp-content/upl…

"The figure for Ethiopia is derived from a sample of 250, 15-year-old immigrants to Israel. The figure for Nigeria is from a sample of 86 adult men & one of 375, 6–13-year olds. The figure for Sierra Leone from a sample of 22, 23-year-old skilled workers & one of 60 adults" 5/n

There is now a newer version of these 'national IQ' estimates by Becker, available online, largely based on Lynn and Vanhanen's updated 2012 dataset. Surely this updated version will have fixed many of the sampling problems of earlier versions, right? 6/n

Nope

The 'national IQ' of Ethiopia - a population of 112 million - is now based on 6 samples, with a total sample size of 707, all are samples of children, and all of which are highly unrepresentative of the country 7/n

3 are samples of Ethiopian immigrants to Israel - not remotely representative;

2 samples are from the same survey conducted in food-insecure districts. Unrepresentative;

The last is from the Young Lives survey (promising, yes?) but uh-oh... 8/n

“Unfortunately in the Ethiopian version, administration of the test ran into difficulties relating to explanation of tasks & time constraints & only about a quarter of the sample—all urban children—have test scores available in the dataset.” Unrepresentative 9/n

Sierra Leone? This population of 7.8m is estimated from 2 samples from the same study in 1966, ages 10-40, one rural & one urban sample of the same ethnic group. Sample size? 119 participants (any demographer reading this has now had to stop & lie down in a darkened room) 10/n

Nigeria? 10 samples here, total sample size of 12,331 - this is better, right?

No. All but one are samples of children

The one adult sample has a sample size of 47 (35 men, 7 living abroad) but no other info on sampling strategy. Wildly unrepresentative 11/n

2 samples involve over-representation of twins (including largest sample of 11,164). Twins are kept in the sample despite having lower cognitive scores.

One is of ‘aggressive’ boys & controls; ‘aggressive’ boys are kept in the sample despite having lower cognitive scores 12/n

2 samples are of private & public schools in same area

2 more samples, boys only, were from ‘modern’ & ‘traditional’ schools in same area

All are unrepresentative as samples were chosen to represent different sub-populations 13/n

One sample is of children attending an outpatient clinic at one hospital; final sample is from a single school grade, from one state, in 1981. Unrepresentative

Extrapolating 'national IQ' estimates from such unrepresentative samples will result in inaccurate, biased data 14/n

I'll throw in one more example, since it's so egregious: the 'national IQ' of Botswana - a population of 2.3m - is estimated from a sample of 140 17-20 year olds from the Batswana ethnic group, sampled from schools in the North West Province of *South Africa* 15/n

Here I've focused on the wholly inadequate samples which go into these 'national IQ' estimates, but many, many other criticisms can, and have been, made. It is very, very depressing that these data are still appearing in peer-reviewed journals 😢 /end