Yes, I know you know this, but it’s so easy to forget!
Yeah, YOU OVER THERE, you with the p-value of 0.0000001 — yes, YOU!! That’s not causation.
2/
No matter how small the p-value for a regression of IQ onto shoe size is, that doesn’t mean that big feet cause smarts!!
It just means that grown-ups tend to have bigger feet and higher IQs than kids.
3/
So, unless you can design your study to uncover causation (very hard to do in most practical settings — the field of causal inference is devoted to understanding the settings in which it is possible), the best you can do is to discover correlations.
Sad but true.
4/
TWO: A P-VALUE IS JUST A TEST OF SAMPLE SIZE.
Read that again — I mean what I said!
If your null hypothesis doesn’t hold (and null hypotheses never hold IRL) then the larger your sample size, the smaller your p-value will tend to be.
5/
If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE.
6/
Why does this matter?
In many contemporary settings (think: the internet), sample sizes are so huge that we can get TINY p-values even when the deviation from the null hypothesis is negligible.
In other words, we can have STATISTICAL significance w/o PRACTICAL significance.
7/
Often, people focus on that tiny p-value, and the fact that the effect is of **literally no practical relevance** is totally lost.
8/
This also means that with a large enough sample size we can reject basically ANY null hypothesis (since the null hypothesis never exactly holds IRL, but it might be “close enough” that the violation of the null hypothesis is not important).
9/
Want to write a paper saying Lucky Charms consumption is correlated w/blood type? W/a large enough sample size, you can get a small p-value. (Provided there’s some super convoluted mechanism with some teeny effect size… which there probably is, b/c IRL null never holds)
10/
THREE: SEEK AND YOU SHALL FIND.
If you look at your data for long enough, you will find something interesting, even if only by chance!
In principle, we know that we need to perform a correction for multiple testing if we conduct a bunch of tests.
11/
But in practice, what if we decide what test(s) to conduct AFTER we look at data? Our p-value will be misleadingly small because we peeked at the data.
Pre-specifying our analysis plan in advance keeps us honest… but in reality, it’s hard to do!!!
12/
That’s it for today. Have a great weekend! 🌞⛱️🩴
13/13
• • •
Missing some Tweet in this thread? You can try to
force a refresh
This new torch lab will make it easier for instructors to teach deep learning, without needing to troubleshoot a classroom full of keras installation errors. 👩🏫💻😭
Thanks to @dfalbel and @zkajdan for this valuable contribution!!! 🙏🙏🙏
My daughter and I have been sitting on the tarmac at SeaTac for 6 hours waiting for our flight to take off.
@alaskaair what is going on?! 6 hours is way too long to have a 7yo sitting on a plane, especially without snacks, water, or updates about flight status
Also, thankful for our n95s, I can’t see a covered nose anywhere
Luckily I brought emergency provisions. This wasn’t special for the flight, I literally never leave the house without an entire produce aisle in my bag, otherwise I get too hangry
The 2nd edition has been in the works for literally 4 years, which -- as a point of reference -- is more than 5x longer than it takes to make a human baby.
It has 50% more material than the original edition, including three all-new chapters! 3/
A lot of people have reached out to check in on me after this article came out --- in which I use a very colorful analogy to describe the current situation that many mothers in academia face during the pandemic 1/ nytimes.com/2021/04/13/hea…
If you reached out, then thank you for the concern. I have been very fortunate: since late Spring 2020, I have had the good fortunate of consistent high-quality childcare for my 3 small children. (Not to mention that I'm senior faculty w/ job security.) 2/
But many mommas haven't had access to high-quality childcare: either due to $$ considerations, or because their family could not tolerate the additional COVID risk associated with childcare, or for a host of other reasons. 3/
I am loving these recent threads from @j_l_godwin on ADHD + academia. Thank you for sharing your experiences, Jessica --- I have learned a lot, and this new knowledge will help me be a better teacher/mentor for students in the future.