Thread by @NoahHaber

Oh gosh, thanks for this opportunity to talk about one of my favorite nerdly science topics:

THE ABSENCE OF EVIDENCE FALLACY!

* before I move on, I want to state upfront that there is, in fact, decent evidence of this, but it's a tricky thing to nail down, which I will address

https://twitter.com/conservmillen/status/1267941393424670721

You see this argument all the time. Hell, I've done this plenty.

"There is no evidence of <insert whatever thing you are saying doesn't exist>"

Often, you'll see it alongside some citation of a paper did not find a statistically significant effect of ... something.

The standard retort against this is that "absence of evidence is not evidence of absence."

This is CRUCIALLY important to understand, but it also doesn't quite go far enough.

Let's start by going over what would have to be true for absence to be evidence.

Absence of evidence being evidence of absence is roughly equivalent to saying:

"If it exists, we would have seen it definitively, with 'evidence'"

OR

"We have already found everything that exists that is relevant to this topic."

Which is (technical term) fucking absurd.

To get a bit more concrete, there have been a few well-known papers out there that found "no evidence" of race-based discrimination in police violence, most notably Fryer 2016/2018.

Let's assume those papers are perfect within the scope of what they measured.

I am going to all caps this: WITHIN THE SCOPE OF WHAT THEY MEASURED.

Here's the thing: science is hard. It isn't magic, and we can't actually measure everything.

This one paper deals (mostly) with a relatively narrow slice of the potential scope of problems.

Specifically, it (mostly) deals with a relatively late stage of police interactions, AFTER the point of stopping and questioning. And at this point, it does not find substantial race based differences in use of force.

I won't get into the nitty gritty here, just concepts.

But that doesn't mean that such differences don't exist, either WITHIN or OUT of it's scope.

Clearly, the out of scope bit makes sense: it can't say anything about things it doesn't say anything about.

Another common example of this is with gender/race and income.

When you "control" for job, income disparities tend to be substantially reduced (often to non-significance).

But job trajectories are the main mechanism for income changes, so you've controlled away the main driver of income changes, you find "no evidence."

See?

But that phrase "non-significance," is ALSO misleading. Non-significant doesn't mean non-existent, but that whatever it is didn't meet an arbitrary threshold for statistical noisiness.

It also doesn't mean that you're measuring the right thing.

This is a rant for another day.

BUT WAIT THERE'S MORE!

I've just used the word "evidence" 11 times thusfar (12 now). And not a once I have I defined what "evidence" is (13).

Because evidence (14) isn't such an easy thing to nail down, and it's very rarely direct and complete.

Is it only "evidence" when one person writes it and 3 other people thought it wasn't so much a total garbage pile that it's ok if we share it with other sciencey folks (aka peer-reviewed publications)?

What if it is, but it isn't EXACTLY about the topic we want?

What about all the evidence that doesn't come in the form of peer reviewed publications?

Don't get me wrong, I fucking LOVE quantitative evidence when we can do it.

I also know we can't always do it.

And there's a lot of non-quanty evidence we usually aren't including.

Sn aside: the existence of evidence is not the existence of GOOD evidence, but that's another lifetime of rants.

When we can arbitrarily define what evidence is and isn't, it's super easy to "absence of evidence" just about anything.

And that tends to get abused to death.

One way it gets abused to death is that you can just say it, and chances are, the person you are saying it to doesn't know better. THEY haven't seen the "evidence" so probably doesn't exist.

And that's ALSO stupid. Because sometimes, it does.

Like here. Ish.

I know enough to know that I am out of my depth on discussing the totality of evidence of race and crime. Ain't my field.

What I DO know is that there is a good amount of discussable evidence out there, of varying levels of usefulness and quality (and that IS my field).

But there's the OTHER danger: mindlessly repeating "evidence of absence is not absence of evidence" as a refutation of the person's argument.

Because what this means isn't that the greater argument is wrong, but merely that that point doesn't add anything to the debate.

In the case of the original, the greater argument is, indeed, wrong for all the reasons discussed above.

These kinds of discussions also tend to go nowhere and because citation-count fishing exercises, so strategically it's not always the best idea to get into that either.

Now I'm no saint here; I've done all of these wrongs aplenty. And chances are, you have too, and agreed with this kind of thing.

But it's damn important to recognize what these things are when they happen. Lives depend on it.

Fin.