This is why most published research is false (as in being non-replicable)
You ask why?
Let’s dive in.
You ask why?
Let’s dive in.
https://twitter.com/fmg_twtr/status/1334884184675012609
1/ You are looking at the distribution of Z-values from medical papers.
Z value tells you how unlikely is your observed data to happen by chance given there is no effect.
Which means, lower the Z-value, the more likely your results are due to noise and not genuine phenomena.
Z value tells you how unlikely is your observed data to happen by chance given there is no effect.
Which means, lower the Z-value, the more likely your results are due to noise and not genuine phenomena.
2/ Notice the two nice bumps around 0?
They correspond to conventional threshold of p<0.05 which a research result must qualify (by convention) to be published.
They correspond to conventional threshold of p<0.05 which a research result must qualify (by convention) to be published.
3/ There are two problems with the graph.
Below this threshold, there are hardly any published research which means all negative results either get mined to push it over the boundary or researchers give up and move to another problem.
Below this threshold, there are hardly any published research which means all negative results either get mined to push it over the boundary or researchers give up and move to another problem.
4/ Researchers are humans too so my guess is that abandoning a project whose results don’t look significant isn’t easy.
The more plausible behaviour is that researchers keep looking at various angles unless a significant result arises.
The more plausible behaviour is that researchers keep looking at various angles unless a significant result arises.
5/ It happens because of “publication bias” of publishing results with p<0.05.
What if you get p=0.07? You think researchers will give up?
No, and that is why you see such a sudden bump in either sides of 0 in the Z-value graph.
What if you get p=0.07? You think researchers will give up?
No, and that is why you see such a sudden bump in either sides of 0 in the Z-value graph.
6/ Results have to be pushed over this boundary because journals demand it.
7/ The second (related) issue is the decay of Z-values beyond the bump.
If all of these are genuine results, you’d expect a flattish curve.
A result with Z-value of 10 should be equally likely as 9.
If all of these are genuine results, you’d expect a flattish curve.
A result with Z-value of 10 should be equally likely as 9.
8/ After all, it’s a test of significance and genuinely significant results can have any level of significance.
It’s baffling to observe that genuinely significant results cluster in significance.
You shouldn’t expect that because your study is done to estimate unknown effects
It’s baffling to observe that genuinely significant results cluster in significance.
You shouldn’t expect that because your study is done to estimate unknown effects
9/ It’s also super interesting to see the peak of positive Z value to be higher than peak of negative Z value.
Why would that be? I don’t know.
Why would that be? I don’t know.
10/ Similarly, the tiny bump on 0 represents the trailblazer journals who’re not afraid to publish negative results.
These journals reward rigorous research by publishing results irrespective of what came out of the research.
These journals reward rigorous research by publishing results irrespective of what came out of the research.
11/ It was fun trying to dissect this.
It’s a field that’s called meta-science, and to read more about it, I suggest starting with this accessible paper.
journals.plos.org/plosmedicine/a…
It’s a field that’s called meta-science, and to read more about it, I suggest starting with this accessible paper.
journals.plos.org/plosmedicine/a…
12/ Oh by the way, the reason it’s non-replicable is because in the subconscious desire to push results over to significance barrier, you may have found random noise that looks like significant data.
Throw a million darts and you’ll find a few that look like an unusual cluster
Throw a million darts and you’ll find a few that look like an unusual cluster
• • •
Missing some Tweet in this thread? You can try to
force a refresh
