,
4 tweets,
2 min read
Read on Twitter
I've seen a growing number of computational social science projects using the genderize.io API to assign F/M gender labels to names in research studies. It's an attractive service to use! But, it's not reliable enough for research projects. Here's why: 1/4
The API returns one of {F,M,-} based on the first name, with '-' meaning "don't know." On 500 names sampled randomly from 20,000 ground-truthed faculty names I have, genderize.io guessed correctly only 34% of the time, and it only made a guess on 50% of queries. 2/4
Here's the full confusion matrix, which reflects a F/M gender imbalance in my empirical data. Only if we ignore the "don't know" column entirely (50% of queries!), does "accuracy" get inflated to 68%, which is still pretty poor. 3/4 
Punchline: genderize.io isn't research-grade, and its errors can't be modeled because it's not open to inspection. My recommendation: use Ethnea+Genni2.0, which is built by academics (but, respect their rate limit request) : abel.lis.illinois.edu/cgi-bin/ethnea… 4/4
