Thread by @hardsci

You guys. We just did a little demo in my research methods class to introduce them to p-values and statistical significance. And I'm super excited at how it went down

The demo was simple. Students pair up. One student, the flipper, flips a coin. The other, the guesser, guesses the result. Flipper tells them right or wrong. Then they switch roles and do it again. Everybody reports their results on their iClicker

Before I show the result, I ask them what they think it'll be. Some say 50-50. Some say not exactly 50-50 because of randomness. Are there other possibilities besides that? We discuss stuff like maybe people are faking-good. Or they can read partner's nonverbals. Or minds. Etc.

So now we're going to create a statistical model of how probable are different outcomes under random guessing & compare our result to that. I show them a barplot of the sampling distribution. I haven't looked at results and I'm totally expecting something boring. But guess what..

IT'S SIGNIFICANT.

Like even Benjamin et al. significant! (p = .002)

60 out of 93 students guessed right

So I'm like, by our decision rule* we are supposed to reject the null. And we talk about how even rare stuff happens - a.k.a. honest false positives. But also, maybe one of those other possibilities actually happened. Maybe y'all have ESP

* alpha = .05 (totally unjustified)

And somebody is like, but wait, shouldn't we do it again?

And on the spot (not planned b/c I was not expecting this result) I'm all YES LET'S RUN A REPLICATION

So we run the replication. And now it's 54% correct, p = .17. And we can talk about statistical flukes, methodological differences (like maybe people misunderstood instructions the first time), and replication

Anyway I'm happy my chintzy little demo gave a surprising result that I could say more interesting things about than I had planned. (Though a little disappointed my students don't have ESP.) Now if only I can replicate it next year...