It’s really is really hard to say!
It’s a strange way of trying to limit false positives that has taken on nearly sacred value.
And its intellectual history is a mess. Like: All research is built around this 5% level... which RA Fisher just plucked from thin air.
It’s a strange way of trying to limit false positives that has taken on nearly sacred value.
And its intellectual history is a mess. Like: All research is built around this 5% level... which RA Fisher just plucked from thin air.
Basically, it’s a lifetime false-positive batting average. If you strictly hold to the 5% level, then in cases where you could make a false positive claim (there is no relationship, but you think there could be), you’ll falsely find an effect ~5% of the time.
That’s it!
That’s it!
It doesn’t say anything about this specific paper you’re reading.
It doesn’t say anything about how often you make false negative errors (when there IS an effect, but you say “nah”).
And that’s before you get to the flagrant corruption that shows how little we get it:
It doesn’t say anything about how often you make false negative errors (when there IS an effect, but you say “nah”).
And that’s before you get to the flagrant corruption that shows how little we get it:
Like: YOU CAN’T PICK YOUR LEVELS AFTER THE FACT.
If a paper has “* means 10%, ** means 5%, *** means 1%”—and they all do!—that. Is. Wong.
You set the level in advance—it’s a career error level—and hold to it. p = 0.000001 or p = 0.049 are IDENTICAL if you set your level at 5%.
If a paper has “* means 10%, ** means 5%, *** means 1%”—and they all do!—that. Is. Wong.
You set the level in advance—it’s a career error level—and hold to it. p = 0.000001 or p = 0.049 are IDENTICAL if you set your level at 5%.
Also, we NEVER TEST THE NULL WE BELIEVE IN.
Literally flipping Popper on his head.
“I’m testing the impact of police on crime. My null is no effect.”
NO IT ISN’T.
So now you’re seeking confirmation, not rejecting the null. We dishonor the idea COMPLETELY.
Here’s the tell:
Literally flipping Popper on his head.
“I’m testing the impact of police on crime. My null is no effect.”
NO IT ISN’T.
So now you’re seeking confirmation, not rejecting the null. We dishonor the idea COMPLETELY.
Here’s the tell:
If we really took Popperian p-value hypo texting seriously, then your regression’s point estimate?
DOESN’T PROVIDE ANY INFORMATION.
All you’ve done is reject the null (that you didn’t believe).
But what comes next in every paper? “My estimate of 0.4 means....”
No. Nope. No.
DOESN’T PROVIDE ANY INFORMATION.
All you’ve done is reject the null (that you didn’t believe).
But what comes next in every paper? “My estimate of 0.4 means....”
No. Nope. No.
If we were really committed to Popper’s approach, my estimate would reject the null of no effect, and then your paper would have to propose SOMETHING ELSE and reject THAT.
But we NEVER do that, bc a no effect null—a null we rarely believe—is the default in EVERY stats program.
But we NEVER do that, bc a no effect null—a null we rarely believe—is the default in EVERY stats program.
The p-value is thus a mess all the way down, even when we CAN explain it.
It has its uses, in certain places, but it def does NOT deserve the gate-keeping power we give it.
It has its uses, in certain places, but it def does NOT deserve the gate-keeping power we give it.
