Thread by @magi_jay

I wrote a piece about rising crime rates & the lack of evidence behind the theory that racial justice protests *cause* these changes via "The Ferguson Effect." In response, some pointed to this working paper as proof of such an effect. There are a lot of issues here. (Thread)

https://twitter.com/magi_jay/status/1399851669647769603

First, let’s be clear about what the very specific causal claim of the “Ferguson Effect.” It’s a complex one, so each step in the chain matters:

I tend to think this chain of causality is implausible, but I will always consider evidence. In the case of this paper, the researchers claim that police killings that go viral AND are accompanied by investigations correlate w/ rising homicides.

If this paper were methodologically sound, it would *still* not show evidence of the Ferguson effect. There are all kinds of reasons why crime might increase or police interactions might decrease after a high-profile case of police-use-of-force.

So even if this paper didn’t have issues, it still wouldn’t be demonstrating evidence for the "Ferguson Effect." That said, I looked through the paper, and there are definitely some issues that jump out at me. The paper got a lot of attention, so I think it's worth discussing

The researchers looked at police killings in 27 incidents. They found a significant increase in homicides after only *5* of the investigations. 1 was followed by a significant decline. In 22 cases, the majority of the data, there were no significant differences in homicide rates.

The authors asserted that these 5 cases were different than all the others because they alone were preceded by a viral event involving the incident. They do not, at this point, specify how they quantify “viral,” which is a problem. They then *only* analyzed these 5 cases.

You should have follow-up Q's when you see people run significance tests, then throw out the majority of their data, then claim the remaining data perfectly correlate with one explanatory variable ("viral," which they haven’t quantified) & therefore they proved their hypothesis.

The next issue is even more important. The researchers said they found 5 instances where violence increased after investigations of police killings that went viral. Going viral + investigation was the supposed causal factor that distinguished these incidents from all others. . .

The researchers found the effect they were looking for after these killings: Freddie Gray (2014), Laquan McDonald (2014), Timothy Thomas (2001), Tyisha Miller (1998), and Michael Brown (2014).

The working paper was distributed in 2020. Do you see any names missing? I do.

Alton Sterling. Philando Castile. Tamir Rice. Eric Garner. To name just a few

All of their deaths were associated with a lot of online attention. All of their deaths were investigated (though not to the extent we would have liked)

Why were they excluded from the analysis?

Is this intentional cherry-picking? I don’t know. But there are real issues with the data. Was Eric Garner excluded because he wasn’t shot, but rather asphyxiated? That doesn’t make much sense, given Freddie Gray, who was not shot, but rather battered in a van, was included.

Why was Tamir Rice excluded? At one point, the authors seem to assert his case is different because it wasn’t investigated by the DOJ. This makes their claim even more convoluted. For the increase in crime to occur, the killing has to go viral & be investigated by the DOJ?

And why was Tyisha Miller’s case included? Certainly her case should be discussed in the context of police-use-of-force, but the claim here involves a causal link between a *viral* event and increases in violence. Miller was killed in 1998. Did her murder go “viral?”

The authors do refer to the fact that they had to limit their analysis to cases where they could obtain data. Could they not obtain complete data from NYC or St Paul/Minn? Maybe not, though I suppose I am dubious.

But, even if they couldn’t obtain these data, a lack of analysis of these high-profile events, among many others, renders their research woefully incomplete.

Researchers need to be explicit about what these limitations mean for their conclusions. &, in this specific case, authors need to “Say Their Names,” such that they acknowledge how excluding the deaths of Rice, Garner, Castille, and so many others seriously limits their analysis

At the end of the day, this paper doesn’t demonstrate the “Ferguson Effect.” There are also issues with post-hoc decisions, data exclusion, retrofitting of hypotheses, etc. . And, importantly, the final conclusion, after all of this, is still fairly clumsy.

That said, even if the evidence *were* strong and the Ferguson effect was affirmed, if police aren’t doing their jobs, that’s on them. It’s not the fault of protesters, viral videos, or DOJ investigations.

But, again, the evidence isn’t pointing in that direction.

Oh, btw, murder rates do not appear to have increased in NYC in the aftermath of Garner's death, using yearly data. Did they spike in the month afterwards? I don't know. But the lack of yearly change + the exclusion of Garner does make me raise an eyebrow about the research.

Here is a link to the paper nber.org/system/files/w…