Thread by @MBarczentewicz

🚨 Now in @ModernLRev (open access): bad statistics led the government to conclude that 'Cart' immigration judicial reviews are ineffective. My computational answer shows how such research should be done and that the govt's conclusion was wrong.
🧵 TLDR in the thread below

The government relied on two analyses. The first came from the Faulks (IRAL) report. In it, the authors made two very basic (and indefensible) errors:

(1) They looked for evidence of successful Cart JRs in a wrong database, which only included a small sample of relevant cases /1

(Hence, they unsurprisingly found only a small number of 'successful' cases.)

(2) They then compared this small number with the total number of all Cart JRs — even though they only looked for successes in a small sample!!! This is how they ended up with the 0.22% ratio. /2

This was rather embarrassing, so the government conducted a second analysis, using better data from internal courts' databases. Unfortunately, they also showed poor research practice:

(1) They disregarded cases that were potentially settled out of court, assuming failure. /3

(2) They didn't address data quality issues in their database (noticed in previous academic research).

(3) Most importantly: they compared the Cart success rate they calculated (3.4%) with success rate in non-Cart cases in the ‘range of 30% to 50%’. Why is this a problem? /4

Because we don't know what is the success rate in judicial review! No one ever managed to find out on a sufficiently representative sample. This is not surprising: researchers have big problems with access to relevant data. But just think about it: /5

We're thinking about reforming judicial review, partially based on its cost and alleged wastefulness, but no one really knows how often claimants succeed! Both allegations of waste and their refutations are based on intuitions and anecdotes, not on sound general data. /6

In my study of effectiveness of Cart JRs, I created a custom dataset of over 42,000 Upper Tribunal (scraping gov.uk pages) and then analysed it using Python and Elasticsearch (see the appendix barczentewicz.com/publication/ca… ) /7

(You can see the JavaScript interface I created to support my analysis in this older thread:) /8

https://twitter.com/MBarczentewicz/status/1382048272601247746

I concluded that ignoring settlements, the success rates are very close – 2.3% (or 3.4% according to the government) for Cart and 3.9% for non-Cart cases in the same period. With settlements the comparison could be less favourable to Cart cases, but we just don't know. /9

What are the lessons?

(1) Legal reforms should be based on sound evidence. Given that key evidence doesn't exist today (success rates!), making necessary data available should be a priority. Hopefully this will change once @UkNatArchives start publishing court data. /10

However, what we need is not just texts of court decisions, but also "meta-data" - ideally more detailed than e.g. JR Case Level data published by MoJ already (e.g. to help address the question of settlements). /11

(2) Evidence accompanying reform proposals should and will be scrutinised. It thus needs to be *trustworthy*. Adhering to principles of reproducible and open research (which neither government analysis did) should be standard - see @turinginst's turing.ac.uk/research/resea… /11

(3) Finally, legal research (also doctrinal research) should be more attentive to the problem of data — e.g. of using a small sample of relevant cases. Possibly, in near future we'll be looking at some of today's (even doctrinal) studies as missing the forrest for the trees. /12

My companion study to the MLR article—forthcoming in Judicial Review—is an example of what we can learn by doing doctrinal research on the kind of court decisions that aren't studied today often, partly due to data access issues. /13 barczentewicz.com/publication/ca…

The MLR paper can be accessed here: onlinelibrary.wiley.com/doi/10.1111/14…