Thread by @MDaware

@WrayCharles @TomVargheseJr maybe tomorrow

@WrayCharles @TomVargheseJr okay so let's say you work for a famous irish brewery and you want to try out some new ingredients and see if the beer is better with them

@WrayCharles @TomVargheseJr you make 2 big batches of beer, one with ingredient A, one with ingredient B

@WrayCharles @TomVargheseJr now, you want to see which is better. but you can't just drink the entire vat of each and say A is better or B is better because that's just too much plus then you wouldn't have any to sell

@WrayCharles @TomVargheseJr so you take a SAMPLE from each and see if they are the same or different

@WrayCharles @TomVargheseJr the problem is, now you don't know whether the differences in sample A and sample B are because you used A+ New Small Batch Hops or some minor, unknown variation in the brewing process, or random chance

@WrayCharles @TomVargheseJr like maybe the entire vats actually taste the same but you caught some from the bottom of A's vat which was a little burnt or the top of B's where it was super fresh

@WrayCharles @TomVargheseJr so even if A and B are actually the same, there's a chance the samples come out a little different.

that's the p-value

@WrayCharles @TomVargheseJr it's possible that vat A and vat B have different actual values and that's why we got different samples

@WrayCharles @TomVargheseJr the p-value is the chance that "nah, the vats are really identical but you just got 2 different samples due to random chance"

@WrayCharles @TomVargheseJr the fundamental problem is WE JUST DON'T KNOW

we know there are 2 main possibilities, and the p value is the chance that the null hypothesis would come up with this answer on it's own, but we still don't know

@WrayCharles @TomVargheseJr @methodsmanmd one of the consequences, of course, is that roughly 5% of the time, when there is no true difference, we interpret that P<0.05 as suggestive of there being a difference. and given the literally thousands of studies published every day...

@WrayCharles @TomVargheseJr @methodsmanmd so that's it. that's p values in a nutshell

last point is that Guinness didn't want Gosset giving away their secret methods for making better beer, so he published under the pseudonym "Student," which is why we call them "Student's t-tests"

@WrayCharles @TomVargheseJr @methodsmanmd for the record I do not pretend to be a complete expert, for this to be exhaustive nor absolutely cover it all perfectly. but I find this an intuitive approach to understanding what a P says (and doesn't). comments welcome!