DEFINITION OF A P-VALUE. Assume your theory is false. The P-VALUE is the probability of getting an outcome as extreme or even more extreme than what you got in your experiment.
THE LOGIC OF THE P-VALUE. Assume my theory is false. The probability of getting extreme results should be very small but I got an extreme result in my experiment. Therefore, I conclude that this is strong evidence that my theory is true. That's the logic of the p-value.
THE P-VALUE IS REASONABLE IN THEORY BUT TRICKY IN PRACTICE. In my opinion, the p-value is just a mathematical version of the way humans think. If we see something that seems unlikely given our beliefs, we often doubt those beliefs. In practice, the p-value can be tricky to use.
THE P-VALUE REQUIRES A GOOD DEFINITION OF WHEN YOUR THEORY IS FALSE. There are usually an infinite number of ways to define a world where your theory is false. P-values often fail when people use overly simplistic mathematical models of the processes that created their data.
If the mismatch between their mathematical models of the world and the actual world is too large then the probabilities we compute can become completely disconnected from reality.
THE P-VALUE MAY REQUIRE AN ACCURATE MODEL OF YOU (THE OBSERVER). The probability of getting the result you got depends on many things. If you sometimes do things like throw out data or repeat measurements then you're part of the system.
Your behavior affects the probability of getting your experimental results. Therefore, to be completely realistic, you need to have an ACCURATE model of your own behavior when you gather and analyze data. This is hard and a big part of why the p-value often fails as a tool.
BY DEFINITION, P-VALUES MUST SOMETIMES BE WRONG. When using p-values, we're working off of probabilities. By logic of the p-value itself, even with perfect use, some of your decisions will be wrong. You have to embrace this if you're going to use the p-values.
Badly defining what it means for your model to be false. Inaccurately modeling the chances of getting your data including your own behaviors. Not treating a p-value as a decision rule that can sometimes be wrong. These factors all contribute to misuse of the p-value in practice.
Hope this cleared some things up for you. Thanks for coming to my p-value TED talk!
• • •
Missing some Tweet in this thread? You can try to
force a refresh
One of the things I hate most about the cult of IQ is it leads to lot of magical thinking about how the brain works. There’s absolutely nothing shameful about relearning things you use to know.
Research shows forgetting is a normal part of human cognition.
The way to combat the natural tendency to forget is to relearn or retrieve the memory at regular intervals which is known as “spaced repetition”.
Relearning takes the strength of the memory back to 100% and the rate of forgetting is slower the next time.
Infographics of this dataset have been kicking around on the internet for years. It is an insult to real scientists everywhere. For every 10 likes, I will post a new ridiculous fact about how fake and ridiculous this "data" is.
They report data on 185 countries but *104* of those numbers (more than half!) are based on *zero* data collected from people from that country. ZERO.
Rather than acknowledge this lack of data, they decided to guessimate based on surrounding countries.
The IQ estimate for Equatorial Guinea was based on kids in a home for developmentally disabled kids living in Spain. Not even their home country. Spain.
People are getting thousands of likes for spreading this misinformation about sex differences. Let me explain why this interpretation of the data is wrong. 🧵
If you think 100% accuracy is too good to be true, trust your instincts.
The version of the model shown in the plot was basically fed the sex of the participants. That’s why it’s achieving 100% accuracy.
When the model was tested on a subset of people from the same dataset that it had *not* seen previously, the accuracy fell to 90%.
I keep seeing this Huberman clip all over my timeline so let’s use it as teachable moment to learn some statistics.
The basic mistake is not taking the people who are already pregnant out of the pool of people who could be pregnant the next month. Of the starting 100, fewer and fewer will remain each month.
It’s a little tedious to keep track of what number of people aren’t yet pregnant on each round, and then take 20% of that, and then add up all the pregnant people in each round.