Thread by @daniela_witten

WHY DID I SAY A P-VALUE IS A TEST OF SAMPLE SIZE?

THREAD 🧵



Suppose we want to test whether the mean of some random variable X is zero. To keep it simple, X1,...,Xn are i.i.d. N(mu, 1). Testing if mu=0.

1/🧵

Now suppose that IRL mu=0.001 (remember- IRL the null hypothesis never exactly holds!)

The z-test for testing mu=0 involves computing

Z=sqrt(n)*Xbar/sigma

and comparing to a N(0,1) distribution. And sigma=1 by my earlier assumption that X_1,...,X_n are i.i.d. N(mu,1).

2/

So Z=sqrt(n)*Xbar.

Suppose n=10^12 so sqrt(n)=1000000.

This means Z=1000000*Xbar.

And Xbar should be somewhere in the ballpark of 0.001 since it's the sample mean of a bunch of observations with mean mu=0.001.

3/

So Z=1000000*Xbar equals something humongous, much much bigger than the any reasonable quantile of a N(0,1) distribution. So we get a small p-value.

If we wanted to be fancy we could compute the actual power to reject the null at any given level alpha (p-value threshold).

4/

But I think intuition is enough for Twitter, so just notice that 1000000*Xbar is around 1000000*0.001=1000 and the probability of a N(0,1) random variable taking on a value of (at least) 1000 is basically 0. So that's a p-value of zero.

Hope this helps clarify!

5/5