Statistical significance and p-values:
We often rely on conventional thresholds of statistical significance. We calculate that p<.05, so statistically significant, right?
However, here are a few things to at least consider first.
Thread.
We often rely on conventional thresholds of statistical significance. We calculate that p<.05, so statistically significant, right?
However, here are a few things to at least consider first.
Thread.
● If you do multiple hypothesis tests, then by chance you're more likely to find some p-values below any threshold, even if the null hypothesis is true.
A team showed, if no corrections for false discovery rate, a dead salmon showed significant brain signal measured by fMRI!
A team showed, if no corrections for false discovery rate, a dead salmon showed significant brain signal measured by fMRI!
● If you have very much statistical power, it turns out that p-values such as 0.04 are, perhaps surprisingly, more likely under the null hypothesis.
Hence some have suggested that you should lower the significance threshold as sample size increases.
daniellakens.blogspot.com/2018/12/testin…
Hence some have suggested that you should lower the significance threshold as sample size increases.
daniellakens.blogspot.com/2018/12/testin…
● In practice, interaction effects need much more stringent significance thresholds to replicate.
You need 16 times the sample size to estimate an interaction effect:
statmodeling.stat.columbia.edu/2018/03/15/nee…
Empirically, interactions rarely replicate:
rpubs.com/Jonatan/intera…
You need 16 times the sample size to estimate an interaction effect:
statmodeling.stat.columbia.edu/2018/03/15/nee…
Empirically, interactions rarely replicate:
rpubs.com/Jonatan/intera…
