Thread by @donmoyn

Interesting case of politicians pushing back on audit studies in UK. My sense is that their response is less about the extra work caused by these studies, and more about being concerned that they are subject to a different form of accountability. theguardian.com/politics/2021/…

If you are unfamiliar, audit studies are designs where researchers send a contact to a a group of actor and vary one salient detail (such as a name indicating gender or race), and then look at differential responses. So they involve a) deception, and b) using the subjects time.

Audit studies have been a really useful social science tool, since they allow us to generate causal estimates of existence and scale of hard-to-detect forms of discrimination in e.g., access to public services, employment, voting.

Audit studies are also cheap and easy to run. So there is a real risk, as Guy correctly notes, that because the costs for researchers for low but non-trivial for subjects, subjects end up getting lots of fake emails.

https://twitter.com/guygrossman/status/1365291790748762115?s=20

So its reasonable to have a high ethical bar for audit studies in IRB review. IMO, researchers should have to make case that a) this is the only way to establish claim, b) its not redundant to prior work, and c) its an important issue, and d) minimize work/risk to subjects.

Here is how co-authors and I thought through some of those ethical concerns in an audit study where we found evidence of discrimination against Muslim families in schools. ajps.org/2020/12/30/the…

The claim that responding to audit studies takes too much resources away from vital needs strains credulity, esp. from MPs who have a high volume of mail & staff to deal with it. Would be a shame if this discouraged a research technique that has shone a light on real problems.

In sum: universities and researchers should be careful about ensuring that audit designs are worth it (and most are in my experience). Similarly, politicians should provide a realistic evidence-based justification for objecting to the technique beyond disliking transparency.